Bug 34682 - Old version of NUnit not compatible with Xamarin.UITest
Summary: Old version of NUnit not compatible with Xamarin.UITest
Alias: None
Product: Installers
Classification: Mono
Component: General ()
Version: unspecified
Hardware: PC Mac OS
: --- normal
Target Milestone: ---
Assignee: Alexis Christoforides
Depends on:
Reported: 2015-10-08 09:31 UTC by Tom Opgenorth
Modified: 2016-01-25 22:10 UTC (History)
6 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Tom Opgenorth 2015-10-08 09:31:50 UTC
It seems that Mono ships with NUnit 2.4.8, which may cause problems when used to run Xamarin.UITests. Our current advice for writing cross platform UITests recommends the use of parameterized TestFixtureAttributes which first made available in NUnit 2.5.3 (http://www.nunit.org/index.php?p=testFixture&r=2.5.3). 

Users that attempt to run UITests at the command line using the default nunit-console that ships will Mono will have their test fail because of the older version of NUnit. As well, UITest itself is only actively tested on recent version of NUnit (NUnit 2.6.3 at the time of writing). Because of this the minimum version of NUnit being recommended to UITest users is 2.6.3 or higher.

See the following relevant issues:

* Forums Post "Command Line UITests not running for PCL?" (http://forums.xamarin.com/discussion/comment/156930)
* Desk case 218144 (https://kb.xamarin.com/agent/case/218144)
* Docs issue 875 (https://github.com/xamarin/documentation/issues/875) 
* UITest issue 683 (https://github.com/xamarin/Xamarin.UITest/issues/683)
Comment 1 Alexis Christoforides 2015-10-08 17:28:39 UTC
I'll work on adding this to the Mac package.

Jo, could you do the same for the Windows package?
Comment 2 Jo Shields 2015-10-08 17:30:54 UTC
Are you talking about including NUnit 2.6 only in the .pkg, but keeping the old 2.4 source in mcs/class ?
Comment 3 Alexis Christoforides 2015-10-08 17:54:14 UTC
AFAIK the API changes in 2.6 would break a lot of tests in mono so upgrading it would be a different project. For now we just need to make installing the mono version optional, and then include 2.6 in the package.

2.6.3 has both binaries and src available (2.6.4 has only binaries):

Comment 4 Miguel de Icaza [MSFT] 2016-01-25 19:49:10 UTC
NUnit as installed by Mono should not have any effect on UITests for Android or iOS, can someone explain why we need to upgrade the NUnit that is part of Mono in this scenario?

I suspect this is not something we have to solve.

It seems like people that want to try to test PCL with a newer NUnit should do so by installing NUnit on their own.  At this point the version that is used by Mono should be considered Mono internal, if only because the API in NUnit broke.

This means that there are two ugly choices ahead of us, and I rather side with "do not break existing code".
Comment 5 Tom Opgenorth 2016-01-25 21:10:32 UTC
Didn't know that upgrading the version of Nunit would be that problematic. 

Probably easiest/best to not break existing code; it's pretty trivial for users to set up their CI servers or command lines to use the correct version of Nunit when running UITests at the command line.

Currently, we do have some docs that can help with the problem:

* The UITest documents are pretty clear that UITest is compatible with NUnit 2.6.3/2.6.4
* There is a guide that explains how to install NUnit 2.6.4 via NuGet: http://developer.xamarin.com/guides/cross-platform/application_fundamentals/installing-nunit-using-nuget/

If users keep running into this problem (i.e. command line test-runner is failing with UITests) we can write another doc to explain the issue and work around.
Comment 6 Miguel de Icaza [MSFT] 2016-01-25 22:10:40 UTC
Ok, with the above information, i say that this is beyond the scope of Mono and should be handled elsewhere.