Bug 46208 - Task.Run called, but not excuted
Summary: Task.Run called, but not excuted
Status: CONFIRMED
Alias: None
Product: Class Libraries
Classification: Mono
Component: System (show other bugs)
Version: unspecified
Hardware: PC Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-11-01 14:39 UTC by manuel
Modified: 2017-10-06 21:03 UTC (History)
7 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Test Case (1.81 KB, application/octet-stream)
2016-11-11 14:49 UTC, manuel
Details
TestCase - Ludovic (17.30 KB, application/zip)
2017-02-16 21:57 UTC, Ludovic Henry
Details
Python http server (1.29 KB, text/x-python-script)
2017-02-16 21:58 UTC, Ludovic Henry
Details

Description manuel 2016-11-01 14:39:33 UTC
Hi

i think we have exposed something similar in the past, however i was not able to find it.
Once in a while our calls to Task.Run do not seem to execute. Our first guess is thread pool starvation with some pending executions doing damages.
When we look, though, to the managed thread list we see 4 threads flagged as threadpool's. If this information is being reported correctly, this means that probably something else is happening.

Note : we do use thread pool heavily through async / await

#############################


 === Xamarin Studio Community ===

Version 6.1.1 (build 17)
Installation UUID: c9aeb0c6-dfdc-4b35-ae00-01c52d42790e
Runtime:
	Mono 4.6.1 (mono-4.6.0-branch-c8sr0/ef43c15) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 406010005

=== NuGet ===

Version: 3.4.3.0

=== Xamarin.Profiler ===

Not Installed

=== Xamarin.Android ===

Version: 7.0.1.3 (Xamarin Studio Community)
Android SDK: /Users/Storyo/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		6.0 (API level 23)

SDK Tools Version: 25.1.2
SDK Platform Tools Version: 24.0.0
SDK Build Tools Version: 23.0.2

Java SDK: /usr
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Xamarin Android Player ===

Version: 0.6.5
Location: /Applications/Xamarin Android Player.app

=== Apple Developer Tools ===

Xcode 8.1 (11544)
Build 8B62

=== Xamarin.Mac ===

Version: 2.10.0.105 (Xamarin Studio Community)

=== Xamarin.iOS ===

Version: 10.2.0.4 (Xamarin Studio Community)
Hash: b638977
Branch: xcode8.1
Build date: 2016-10-25 14:38:48-0400

=== Build Information ===

Release ID: 601010017
Git revision: 44d481a9be9cf2bf7c01bfb99c59b77dfa08e712
Build date: 2016-10-25 14:27:24-04
Xamarin addins: 19cdb28081bf28d9688698030abb96e04e391f69
Build lane: monodevelop-lion-cycle8-sr0

=== Operating System ===

Mac OS X 10.12.1
Darwin Manuels-MacBook-Pro.local 16.1.0 Darwin Kernel Version 16.1.0
    Thu Oct 13 21:26:57 PDT 2016
    root:xnu-3789.21.3~60/RELEASE_X86_64 x86_64
Comment 1 Timothy Risi 2016-11-01 17:31:46 UTC
Hi Manual,

Could you provide a test case for us to reproduce the issue with to help track down what might be happening?
Comment 2 manuel 2016-11-02 00:16:05 UTC
Timothy, i don't think this will be easy. We do have a quite complex scenario using the thread pool and it looks like some race condition.
We'll start testing our new version more deeply and try to figure out some pattern.

Manuel
Comment 3 manuel 2016-11-06 05:47:46 UTC
Guys i was trying to find some pattern. During the tests, i did try the code bellow, which creates the situation that we described. Change max threads to 2 is a bit drastic but i think we are experiencing something similar at a larger scale.
The only way i see this happening is wc.GetStreamAsync or stream.CopyToAsync(mStream) doing some operation that never ends and that blocks the thread from thread pool.
However, we did experience this with Thread pool default max and mins and when we saw the number of threads from the thread pool running we were only able to see 3 or 4 in XS after pausing execution.

	int worker;
	int completionPort;

	ThreadPool.GetMaxThreads(out worker, out completionPort);
	ThreadPool.SetMaxThreads(2, completionPort);

	Task[] parallelTasks = new Task[20];
	for (int i = 0; i < parallelTasks.Length; i++)
	{
		var i_ = i;

		parallelTasks[i] = Task.Run(async () => {
			try
			{
				using (var wc = new HttpClient())
				{
					using (var stream = await wc.GetStreamAsync("URL))
					{
						using (var mStream = new MemoryStream())
						{
							await stream.CopyToAsync(mStream);
							stream.Close();
							Debug.WriteLine($"Task {i_} done");
						}
					}

				}
			}
			catch (Exception ex)
			{

			}
		});
	}
Comment 4 manuel 2016-11-08 16:34:56 UTC
ping
Comment 5 manuel 2016-11-11 10:17:23 UTC
Guys, any idea of what is happening? 
At least if it is HttpClient specific or is something with the ThreadPool itself.
Comment 6 manuel 2016-11-11 11:11:49 UTC
Just 1 more comment

if we put the following code we end up having a situation with worker = 2 and if we call Task.Run, it does not execute the code. When the app gets into this state, it will never recover and ThreadPool.GetAvailableThreads will always return 2.


	var t = new Thread((obj) =>
	{
		while (true)
		{
			ThreadPool.GetAvailableThreads(out worker, out completionPort);

			Thread.Sleep(10000);
		}
	});

	t.Start();

Having said this, i'm strongly convinced there is an issue with ThreadPool, which is critical for us right now.
Comment 7 Marek Safar 2016-11-11 11:50:12 UTC
I tries your latest code on iPad / Debug and it prints it gives me 200 available threads.
Comment 8 manuel 2016-11-11 11:54:42 UTC
Marek, you have to also call the first part that is doing http requests and also note that you have to specify some url in that code.
We are forcing maximum number of threads to 2. Returning 200 is the default value.
Comment 9 manuel 2016-11-11 12:13:10 UTC
just 1 comment more. This is happening on the device. Not sure about the simulator.
Comment 10 Marek Safar 2016-11-11 12:51:53 UTC
Could you put together complete sample which you think exposes the issue?
Comment 11 manuel 2016-11-11 14:49:28 UTC
Created attachment 18429 [details]
Test Case

I did try to point do Xamarin's web site, but was not able to reproduce it. With the Portuguese newspaper web site it is happening to me after a few calls. Probably they change http result after a few consecutive requests.
Behaviour that i'm seeing is the following. After a few requests we see worker = 2, QueueUserWorkItem is called and "Ping" will not be written anymore.

ThreadPool.GetAvailableThreads(out worker, out completionPort);

if (worker == 2)
{
	ThreadPool.QueueUserWorkItem((state) =>
	{
		Debug.WriteLine("Ping");
	});
}

P.S.: I'll send you the web site in private
Comment 12 manuel 2016-11-14 11:55:31 UTC
Marek, did you see the issue occurring?
Comment 13 Timothy Risi 2016-12-19 18:42:24 UTC
@Marek did you get the info to test this out?
Comment 14 Vincent Dondain [MSFT] 2017-01-23 21:41:34 UTC
Hi, I created a full iOS test case to reproduce this easily.

https://www.dropbox.com/s/peklr7jgmz627lm/TestCaseBug46208.zip?dl=0

You have to try that on a physical device and after a little while it will stop executing the http requests.

This looks like a thread pool bug so assigning Ludo and moving it to runtime.
Comment 15 manuel 2017-02-14 16:03:28 UTC
Guys, any idea?
Comment 16 Ludovic Henry 2017-02-16 21:57:36 UTC
Created attachment 19853 [details]
TestCase - Ludovic
Comment 17 Ludovic Henry 2017-02-16 21:58:54 UTC
Created attachment 19854 [details]
Python http server

run with `python httpserver.py 8000` or any other port you want it to listen to, and modify Main.cs to point to your ip address and port
Comment 18 Ludovic Henry 2017-02-16 21:59:31 UTC
I reproduced on iPad, and there are 2 issues:
 - the first is with the sample you sent us: Task.Wait on an async/await in a Task.Run will only wait for the code until the first await, so you will not have only 20 concurrent requests, but the number will continue growing.
 - the second is with the HTTP stack: I changed the testcase and run it against a pretty dumb local python http server (it's threaded and synchronously return an empty 200 HTTP response). It works great with wrk/ab - 2000req/s - but hangs on the first 20 requests from mono.

As per point 2, this is clearly something wrong with the HTTP stack, so reassigning to you Marek, and uploading the XS solution and http server I used to reproduce.

Note You need to log in before you can comment on or make changes to this bug.