Bug 59971 - Xamarin.UITest projects are broken in 15.4 previews
Summary: Xamarin.UITest projects are broken in 15.4 previews
Status: RESOLVED DUPLICATE of bug 59302
Alias: None
Product: Xamarin Studio
Classification: Desktop
Component: Test Cloud (show other bugs)
Version: 7.2 (d15-4)
Hardware: Macintosh Mac OS
: Highest blocker
Target Milestone: 15.4
Assignee: Matt Ward
URL:
Depends on: 59302
Blocks:
  Show dependency tree
 
Reported: 2017-10-04 20:55 UTC by David Hathaway
Modified: 2017-10-09 21:34 UTC (History)
2 users (show)

See Also:
Tags:
Is this bug a regression?: Yes
Last known good build: 15.3 (i.e. 7.1.5 (build 2) + Mono 5.2.0.224)


Attachments

Description David Hathaway 2017-10-04 20:55:55 UTC
Running/Debugging of Xamarin.UITest projects is broken in VSMac 15.4 previews.  The issue appears to be caused by an underlying Mono change (see https://bugzilla.xamarin.com/show_bug.cgi?id=59302).  Not 100% sure, but this is manifesting itself in VSMac users not being able to run/debug Xamarin.UITest projects.
Comment 1 Matt Ward 2017-10-05 09:28:00 UTC
Whilst I do not have exactly the same bits as the current beta channel installed I can repro this error with the following repro steps:

1) Create a new iOS - App - WebView.
2) Update Xamarin.TestCloud.Agent NuGet package in iOS project to latest stable. Currently version 0.21.1
3) Update Xamarin.UITest NuGet package in UITests project to latest stable. Currently 2.1.3
4) Build the solution and run the tests from the Unit Tests window.

Expected result:

App opens in simulator and tests run successfully.

Actual result:

App opens in simulator but tests fail with:

ArgumentOutOfRangeException : Length cannot be less than zero.
Parameter name: length

https://gist.github.com/mrward/e112e6684956af95b11cd576cb3c6b4d


You can reproduce the error from the command line just using nunit-console once the path to the .app is added to the StartupApp call:

ConfigureApp.iOS.AppBundle("path/to/test.app").StartApp();
Comment 2 Matt Ward 2017-10-05 10:09:49 UTC
Tried a few different Mono versions and ran the UITests from the command line:

Mono 5.2.0.224 (d15-3/14f2c81) - OK
Mono 5.4.0.73 (2017-06/2f1b9e8) - Fail
Mono 5.4.0.167 (2017-06/6b8abfeb7cc) - Fail
Mono 5.4.0.201 (2017-06/71277e78f6e - Fail
Mono 5.4.0.204 (2017-06/cb4c3ca997f) - Fail
Mono 5.8.0.0 (2017-10/8a2a0628624) - OK
Comment 3 Brendan Zagaeski (Xamarin Support) 2017-10-06 01:25:41 UTC
## Additional testing to answer what fixed the issue

The fix that is present in Mono 5.8 is also present in Mono 5.6.  I performed a bisect using the available CI builds, and found that one of the following 2 commits fixed the issue:

https://github.com/mono/mono/commit/b08c54265b20d914bd3a5f1c0f7a8ad606411c6e "Bump API snapshot submodule"
https://github.com/mono/mono/commit/efc7e54e149d5b4642bc3b89d7d963c98ec84561 "Bump CoreFX"

"Bump CoreFX" is likely the fix (corefx does appear in the stack trace), but unfortunately that bump is a large change, so backporting it would carry risks.

I can attempt a similar bisect to find the point of breakage tomorrow.  My suspicion would be that it's another earlier CoreFX bump.
Comment 4 Brendan Zagaeski (Xamarin Support) 2017-10-09 21:34:37 UTC
I will mark this bug as a duplicate of Bug 59302 to consolidate tracking of the issue.  I will leave this bug categorized under the "Xamarin Studio > Test Cloud" component to help with searchability in case any users search are searching for the issue primarily in the context of Test Cloud.

*** This bug has been marked as a duplicate of bug 59302 ***

Note You need to log in before you can comment on or make changes to this bug.