Bug 59971 - Xamarin.UITest projects are broken in 15.4 previews
Summary: Xamarin.UITest projects are broken in 15.4 previews
Status: RESOLVED DUPLICATE of bug 59302
Alias: None
Product: Xamarin Studio
Classification: Desktop
Component: Test Cloud (show other bugs)
Version: 7.2 (d15-4)
Hardware: Macintosh Mac OS
: Highest blocker
Target Milestone: 15.4
Assignee: Matt Ward
URL:
Depends on: 59302
Blocks:
  Show dependency tree
 
Reported: 2017-10-04 20:55 UTC by David Hathaway
Modified: 2017-10-09 21:34 UTC (History)
2 users (show)

Tags:
Is this bug a regression?: Yes
Last known good build: 15.3 (i.e. 7.1.5 (build 2) + Mono 5.2.0.224)

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED DUPLICATE of bug 59302

Description David Hathaway 2017-10-04 20:55:55 UTC
Running/Debugging of Xamarin.UITest projects is broken in VSMac 15.4 previews.  The issue appears to be caused by an underlying Mono change (see https://bugzilla.xamarin.com/show_bug.cgi?id=59302).  Not 100% sure, but this is manifesting itself in VSMac users not being able to run/debug Xamarin.UITest projects.
Comment 1 Matt Ward 2017-10-05 09:28:00 UTC
Whilst I do not have exactly the same bits as the current beta channel installed I can repro this error with the following repro steps:

1) Create a new iOS - App - WebView.
2) Update Xamarin.TestCloud.Agent NuGet package in iOS project to latest stable. Currently version 0.21.1
3) Update Xamarin.UITest NuGet package in UITests project to latest stable. Currently 2.1.3
4) Build the solution and run the tests from the Unit Tests window.

Expected result:

App opens in simulator and tests run successfully.

Actual result:

App opens in simulator but tests fail with:

ArgumentOutOfRangeException : Length cannot be less than zero.
Parameter name: length

https://gist.github.com/mrward/e112e6684956af95b11cd576cb3c6b4d


You can reproduce the error from the command line just using nunit-console once the path to the .app is added to the StartupApp call:

ConfigureApp.iOS.AppBundle("path/to/test.app").StartApp();
Comment 2 Matt Ward 2017-10-05 10:09:49 UTC
Tried a few different Mono versions and ran the UITests from the command line:

Mono 5.2.0.224 (d15-3/14f2c81) - OK
Mono 5.4.0.73 (2017-06/2f1b9e8) - Fail
Mono 5.4.0.167 (2017-06/6b8abfeb7cc) - Fail
Mono 5.4.0.201 (2017-06/71277e78f6e - Fail
Mono 5.4.0.204 (2017-06/cb4c3ca997f) - Fail
Mono 5.8.0.0 (2017-10/8a2a0628624) - OK
Comment 3 Brendan Zagaeski (Xamarin Team, assistant) 2017-10-06 01:25:41 UTC
## Additional testing to answer what fixed the issue

The fix that is present in Mono 5.8 is also present in Mono 5.6.  I performed a bisect using the available CI builds, and found that one of the following 2 commits fixed the issue:

https://github.com/mono/mono/commit/b08c54265b20d914bd3a5f1c0f7a8ad606411c6e "Bump API snapshot submodule"
https://github.com/mono/mono/commit/efc7e54e149d5b4642bc3b89d7d963c98ec84561 "Bump CoreFX"

"Bump CoreFX" is likely the fix (corefx does appear in the stack trace), but unfortunately that bump is a large change, so backporting it would carry risks.

I can attempt a similar bisect to find the point of breakage tomorrow.  My suspicion would be that it's another earlier CoreFX bump.
Comment 4 Brendan Zagaeski (Xamarin Team, assistant) 2017-10-09 21:34:37 UTC
I will mark this bug as a duplicate of Bug 59302 to consolidate tracking of the issue.  I will leave this bug categorized under the "Xamarin Studio > Test Cloud" component to help with searchability in case any users search are searching for the issue primarily in the context of Test Cloud.

*** This bug has been marked as a duplicate of bug 59302 ***