Bug 15104 - Nested async tests freeze Touch.Unit test runner
Summary: Nested async tests freeze Touch.Unit test runner
Alias: None
Product: iOS
Classification: Xamarin
Component: Tools ()
Version: 7.0.0.x
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: 7.2.0 (iOS 7.1)
Assignee: Bugzilla
Depends on:
Reported: 2013-10-01 15:15 UTC by Greg Shackles
Modified: 2015-03-09 12:12 UTC (History)
5 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Greg Shackles 2013-10-01 15:15:58 UTC
I'm running into some cases where test cases that involve nested async calls are resulting in the Touch.Unit runner freezing up. Here's a simple example of a test that freezes it:

	public async Task Repro()
		await Task.Run(async() =>
			await Task.Delay(1000);

		Assert.AreEqual(1, 1);

When I run the app and hit Run Everything, it freezes up. Here is the log output:

2013-10-01 15:10:17.386 AsyncTestRepro[8915:1503] MonoTouch: Socket error while connecting to MonoDevelop on Connection refused
2013-10-01 15:10:19.080 AsyncTestRepro[8915:a0b] [Runner executing:	Run Everything]
2013-10-01 15:10:19.081 AsyncTestRepro[8915:a0b] [MonoTouch Version:	7.0.1]
2013-10-01 15:10:19.081 AsyncTestRepro[8915:a0b] [iPhone Simulator:	iPhone OS v7.0]
2013-10-01 15:10:19.082 AsyncTestRepro[8915:a0b] [Device Name:	iPhone Simulator]
2013-10-01 15:10:19.083 AsyncTestRepro[8915:a0b] [Device UDID:	FFFFFFFF4AC64CA0B9534A47A9900ABA8ECA6214]
2013-10-01 15:10:19.084 AsyncTestRepro[8915:a0b] [Device Locale:	en_US]
2013-10-01 15:10:19.090 AsyncTestRepro[8915:a0b] [Device Date/Time:	10/1/2013 3:10:19 PM]
2013-10-01 15:10:19.091 AsyncTestRepro[8915:a0b] [Bundle:	com.yourcompany.asynctestrepro]
2013-10-01 15:10:19.096 AsyncTestRepro[8915:a0b] AsyncTestRepro
2013-10-01 15:10:19.096 AsyncTestRepro[8915:a0b] ReproTests


Xamarin Studio 4.0.12
Xamarin.iOS (Business Edition)
Mono 3.2.3 ((no/8d3b4b7)
Comment 1 Sebastien Pouliot 2013-10-06 12:18:34 UTC
The SynchronizationContext that is used with NUnitLite when doing async work does not play well with the UIKitSynchronizationContext (i.e. you end up waiting on the thread scheduled to do the work).

That's something I want to fix with the update to NUnitLite 1.0 - but that won't be available in XI until it's done (and well tested). You might want to track the githut repo until then (fix will be available there, to test, before new XI versions).
Comment 2 Sebastien Pouliot 2013-10-06 12:48:15 UTC
Fixed in Touch.Unit / master 16062835e1e9ba01f4fb464f918db2e03c8ccb34

Like I said it might take a while (i.e. not the next maintenance release) before this gets into XI. Look for release notes stating NUnitLite 1.0 support :-)
Comment 3 Sebastien Pouliot 2013-12-09 19:16:01 UTC
You might have noticed that I had to revert my original fix - as it broke other tests. I do have a workaround in mind.
Comment 4 Sebastien Pouliot 2013-12-09 20:32:11 UTC
Fixed in master 392c3611b11612894e9b4cfb5ed50b48e8de0cf9

Tests that cannot be executed from the main thread can use the [Timeout] attribute (e.g. with Int32.MaxValue) to have the nunit framework run it into a separate thread.

That keeps the default Touch.Unit runner behaviour of using the main (UI) thread - and let you mark any test that you need to execute on a different thread.
Comment 5 PJ 2013-12-11 18:45:47 UTC
This fix is planned to be released with Xamarin.iOS 7.0.6, which should hit the beta channel before December 23rd.
Comment 6 Sadik Ali 2013-12-18 07:26:59 UTC
I have checked this issue with following steps and we are getting the same behaviour:

1. Created iOS unit test project.
2. Added above test case into "Tests.cs" file.
3. Debug test on simulator.
4. run created test "repo" and observed that test get freeze

Refer screen shot :http://screencast.com/t/IR7d3RC9h

Checked With:
All Mac
XS 4.2.3 (build 20)
Comment 7 Sebastien Pouliot 2013-12-18 08:29:26 UTC
The fix is in Touch.Unit version 1.0. Xamarin.iOS ships 7.0.x ships version 0.9. Version 1.0 ships with XI 7.1.x (alpha). Changing milestone.

Also you did not follow the requirement stated in comment #4, which is adding a [Timeout (Int32.MaxValue)] attribute to test requiring to be run on a background thread.
Comment 8 Greg Shackles 2013-12-18 09:31:17 UTC
Does that mean that any async test would need to have that attribute then? That is less than ideal, since that makes it different from other platforms that can run the tests as-is. If that's the case, would it be possible to just imply that attribute if a test is marked as async, or returns a Task?

I also realize this could just be an NUnitLite vs NUnit difference in general, rather than a runner problem, so if that's the case I understand and the attribute  is easier to justify.
Comment 9 Sebastien Pouliot 2013-12-18 09:46:50 UTC
> any async test

No, just the ones that assume they run on a background thread. That's not very common (e.g. one case in all the mono unit tests).

> that makes it different from other platforms

Not really. The attribute is part of the NUnit.Framework - and it's the only one in NUnitLite that allows control on thread being used (more control exists in the full NUnit framework).

OTOH adding the attribute makes your test more similar (not different) that other platforms (see below for why) and it's precense won't really affect other platforms.

> would it be possible to just imply
> that attribute if a test is marked as async, or returns a Task?

Not without forking NUnitLite and make some rather big changes (it's not under the runner control). That would make it hard to update to new versions and likely cause other compatibility issues.

> rather than a runner problem

It's a runner difference. Most people does not realize it but many runners run the tests in a background thread (by default). OTOH Touch.Unit does not because a lot of iOS API requires to be executed from the main thread (and it's _much_ easier to get things into the background that getting them back on the main thread).
Comment 10 Greg Shackles 2013-12-18 09:56:11 UTC
Sorry for the misunderstanding, and thanks for the quick detailed response! 

I'm still a little confused on the first part, about whether async tests need that attribute now. Taking my original test case as an example, it just awaits a delay. Originally this caused the UI thread to hang, so I'm guessing that means it should get that attribute now? If that's the case, I'm not sure I understand when there'd be an async test that didn't require it, since this is a very simple example of async. Am I just missing something entirely here?
Comment 11 Sadik Ali 2014-01-30 07:37:56 UTC
I have checked this issue, I added attributes "[Timeout (Int32.MaxValue)]" in test method. Debug application on simulator and run all test successfully.

Screen cast: http://screencast.com/t/3Aka2Zgkb

Checked With:

All Mac
XS 4.2.3 (build 51)
Comment 12 falz.x1 2014-09-04 08:17:27 UTC
Adding the Timeout attribute does not really help.

As you can see in the above screenshot (http://screencast.com/t/3Aka2Zgkb), yes the Repo test reports successful but with 0 assertions - the assertions are never made.
Comment 13 Patrick Lind 2015-03-09 12:04:14 UTC
I'm still having this issue... In Xamarin Studio I have something like this:

		public async void User_Should_Be_Logged_In_When_Valid_Login_Credentials()
			// Arrange
			Password = "PASSWORD";
			UserName = "TEST";

			var loginUser = new User
				Name = "Test User",
				Initials = "TU",
				UserName = "TEST"

			loginService.Setup(s => s.Authenticate(UserName, Password));
			loginService.SetupGet(s => s.CurrentUser).Returns(loginUser);
			loginService.SetupGet(s => s.IsAuthenticated).Returns(true);

			// Act
Freezes ->	await loginViewModel.ExecuteLoginUserCommand();

			// Assert - User is logged in and we navigate to the appropriate view model.
			Assert.AreEqual(1, MockDispatcher.Requests.Count);
			Assert.AreEqual(typeof(MainViewModel), MockDispatcher.Requests[0].ViewModelType);

In my ExecuteLoginUserCommand I am doing something like this:

                        IsBusy = true;
				await Task.Run(() => loginService.Authenticate(UserName, Password), cancellationToken);
			catch (TaskCanceledException e)
				IsBusy = false;

Anyone else having a similar problem?  I'm on Xamarin Studio version 5.7.2 on Mac.
Comment 14 Patrick Lind 2015-03-09 12:12:21 UTC
Ok so the issue here is that my await Task.Run code was throwing an exception.

When I do:

await loginService.Authenticate(UserName, Password) 

It reveals the exception now.  So the real problem is different than I thought... NUnit is swallowing async exceptions and just freezing.  If I correct the issue that caused the exception, the original code works.