Bug 7965

Summary: Regression in 3.0 (System.Threading.Tasks): InvalidOperationException "The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled. IsTerminating is set to True"
Product: [Mono] Class Libraries Reporter: Andres G. Aragoneses <knocte>
Component: mscorlibAssignee: Jérémie Laval <jeremie.laval>
Status: RESOLVED FIXED    
Severity: normal CC: jeremie.laval, masafa, mono-bugs+mono
Priority: ---    
Version: unspecified   
Target Milestone: Untriaged   
Hardware: PC   
OS: Linux   
Tags: Is this bug a regression?: ---
Last known good build:

Description Andres G. Aragoneses 2012-10-24 08:52:18 UTC
We have detected our AcceptanceTests failing if we upgrade to Mono 3.0.

I still haven't reduced this to a simple testcase, but thought that putting here the stacktrace early on may help somehow. BTW, we are not using any async APIs or C#5.

[The regression is somewhere between the commit 551ca84 (labeled as "2.11.4", but not necessarily the same tag) and the version based on the tarball found in http://origin-download.mono-project.com/sources/mono/mono-3.0.0.tar.bz2.

Stacktrace is (caputed by xsp4):

Listening on port: 8000 (non-secure)
Hit Return to stop the server.
Handling exception type InvalidOperationException
Message is The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
IsTerminating is set to True
System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
  at System.Threading.Tasks.TaskCompletionSource`1[System.Object].ThrowInvalidException () [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Object].SetException (IEnumerable`1 exceptions) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Object].SetException (System.Exception exception) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1[System.Object].InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1 tcs, System.Func`2 endMethod, IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1+<FromAsyncBeginEnd>c__AnonStorey2D`3[System.Object,System.Byte[],System.Int32,System.Int32].<>m__33 (IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Net.WebAsyncResult.CB (System.Object unused) [0x00000] in <filename unknown>:0 
Handling exception type InvalidOperationException
Message is The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
IsTerminating is set to True
System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (IEnumerable`1 exceptions) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (System.Exception exception) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1[System.Int32].InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1 tcs, System.Func`2 endMethod, IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1+<FromAsyncBeginEnd>c__AnonStorey2D`3[System.Int32,System.Byte[],System.Int32,System.Int32].<>m__33 (IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Net.WebAsyncResult.CB (System.Object unused) [0x00000] in <filename unknown>:0 
Handling exception type InvalidOperationException
Message is The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
IsTerminating is set to True
System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (IEnumerable`1 exceptions) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (System.Exception exception) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1[System.Int32].InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1 tcs, System.Func`2 endMethod, IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Threading.Tasks.TaskFactory`1+<FromAsyncBeginEnd>c__AnonStorey2D`3[System.Int32,System.Byte[],System.Int32,System.Int32].<>m__33 (IAsyncResult l) [0x00000] in <filename unknown>:0 
  at System.Net.WebAsyncResult.CB (System.Object unused) [0x00000] in <filename unknown>:0 


PS: I recommend you to create the "3.0" version in Bugzilla :)
Comment 1 Andres G. Aragoneses 2012-10-24 09:10:13 UTC
(In reply to comment #0)
> BTW, we are not using any async APIs or C#5.

Forgot to say, this a ServiceStack-based solution, which may use async APIs.
Comment 2 Andres G. Aragoneses 2012-10-30 12:23:49 UTC
Got a stacktrace now with line numbers:

[0xb4d0cb70:] EXCEPTION handling: System.ArgumentNullException: null handle
Parameter name: waitHandles

"Threadpool worker" tid=0x0xb4d0cb70 this=0x0x569a0 thread handle 0x41f state : not waiting owns ()
  at System.Threading.WaitHandle.CheckArray (System.Threading.WaitHandle[],bool) [0x00085] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading/WaitHandle.cs:82
  at System.Threading.WaitHandle.WaitAny (System.Threading.WaitHandle[],System.TimeSpan,bool) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading/WaitHandle.cs:215
  at System.Threading.RegisteredWaitHandle.Wait (object) [0x00019] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading/RegisteredWaitHandle.cs:74
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
[0xb4d0cb70:] EXCEPTION handling: System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.

"Threadpool worker" tid=0x0xb4d0cb70 this=0x0x569a0 thread handle 0x41f state : not waiting owns ()
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetResult (int) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:86
  at System.Threading.Tasks.TaskFactory`1<int>.InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1<int>,System.Func`2<System.IAsyncResult, int>,System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:461
  at System.Threading.Tasks.TaskFactory`1/<FromAsyncBeginEnd>c__AnonStorey2D`3<int, byte[], int, int>.<>m__33 (System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443
  at System.Net.WebAsyncResult.CB (object) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
[0xb4d0cb70:] EXCEPTION handling: System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.

"Threadpool worker" tid=0x0xb4d0cb70 this=0x0x569a0 thread handle 0x41f state : not waiting owns ()
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetException (System.Collections.Generic.IEnumerable`1<System.Exception>) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:80
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetException (System.Exception) [0x00011] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:74
  at System.Threading.Tasks.TaskFactory`1<int>.InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1<int>,System.Func`2<System.IAsyncResult, int>,System.IAsyncResult) [0x0001f] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:465
  at System.Threading.Tasks.TaskFactory`1/<FromAsyncBeginEnd>c__AnonStorey2D`3<int, byte[], int, int>.<>m__33 (System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443
  at System.Net.WebAsyncResult.CB (object) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
Handling exception type InvalidOperationException
Message is The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
IsTerminating is set to True
System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (IEnumerable`1 exceptions) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:80 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (System.Exception exception) [0x00011] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:74 
  at System.Threading.Tasks.TaskFactory`1[System.Int32].InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1 tcs, System.Func`2 endMethod, IAsyncResult l) [0x0001f] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:465 
  at System.Threading.Tasks.TaskFactory`1+<FromAsyncBeginEnd>c__AnonStorey2D`3[System.Int32,System.Byte[],System.Int32,System.Int32].<>m__33 (IAsyncResult l) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443 
  at System.Net.WebAsyncResult.CB (System.Object unused) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148 
[0xb4d0cb70:] EXCEPTION handling: System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.

"Threadpool worker" tid=0x0xb4d0cb70 this=0x0x569a0 thread handle 0x41f state : not waiting owns ()
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetResult (int) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:86
  at System.Threading.Tasks.TaskFactory`1<int>.InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1<int>,System.Func`2<System.IAsyncResult, int>,System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:461
  at System.Threading.Tasks.TaskFactory`1/<FromAsyncBeginEnd>c__AnonStorey2D`3<int, byte[], int, int>.<>m__33 (System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443
  at System.Net.WebAsyncResult.CB (object) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
[0xb4d0cb70:] EXCEPTION handling: System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.

"Threadpool worker" tid=0x0xb4d0cb70 this=0x0x569a0 thread handle 0x41f state : not waiting owns ()
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetException (System.Collections.Generic.IEnumerable`1<System.Exception>) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:80
  at System.Threading.Tasks.TaskCompletionSource`1<int>.SetException (System.Exception) [0x00011] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:74
  at System.Threading.Tasks.TaskFactory`1<int>.InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1<int>,System.Func`2<System.IAsyncResult, int>,System.IAsyncResult) [0x0001f] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:465
  at System.Threading.Tasks.TaskFactory`1/<FromAsyncBeginEnd>c__AnonStorey2D`3<int, byte[], int, int>.<>m__33 (System.IAsyncResult) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443
  at System.Net.WebAsyncResult.CB (object) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
Handling exception type InvalidOperationException
Message is The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
IsTerminating is set to True
System.InvalidOperationException: The underlying Task is already in one of the three final states: RanToCompletion, Faulted, or Canceled.
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (IEnumerable`1 exceptions) [0x0000c] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:80 
  at System.Threading.Tasks.TaskCompletionSource`1[System.Int32].SetException (System.Exception exception) [0x00011] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskCompletionSource.cs:74 
  at System.Threading.Tasks.TaskFactory`1[System.Int32].InnerInvoke (System.Threading.Tasks.TaskCompletionSource`1 tcs, System.Func`2 endMethod, IAsyncResult l) [0x0001f] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:465 
  at System.Threading.Tasks.TaskFactory`1+<FromAsyncBeginEnd>c__AnonStorey2D`3[System.Int32,System.Byte[],System.Int32,System.Int32].<>m__33 (IAsyncResult l) [0x00000] in /home/dev/mono-3.0.0/mcs/class/corlib/System.Threading.Tasks/TaskFactory_T.cs:443 
  at System.Net.WebAsyncResult.CB (System.Object unused) [0x00000] in /home/dev/mono-3.0.0/mcs/class/System/System.Net/WebAsyncResult.cs:148 


(This is running the 3.0 tarball)
Comment 3 Jérémie Laval 2012-11-16 06:40:28 UTC
The regression might be in WebAsyncResult itself but in the meantime I have committed a safeguard in 94f551d46357c4a71f0108169ac65a3615cda686 (master) that should workaround the issue.
Comment 4 Andres G. Aragoneses 2012-11-16 07:03:11 UTC
Actually, I saw these 3 commits which happened before yours (https://github.com/mono/mono/commit/94f551d4) which are System.Threading.Taskish related:

https://github.com/mono/mono/commit/7c3708d2

https://github.com/mono/mono/commit/e3bd9da8

https://github.com/mono/mono/commit/0cf33093

And I thought they could be related to this. I then tested master branch 2 days after them (after the most recent one), and I didn't get the exception here anymore. But it didn't solve my problem completely (somehow it deadlocks).

So as I also recently saw a commit about a deadlock from Rodrigo: 
https://github.com/mono/mono/commit/47dd377f8587b466475ac5a6cf548f49b9394d0d

I was wondering if that could fix it, but it didn't.

So I'll try master with your fix and report back. Thanks very much.
Comment 5 Jérémie Laval 2012-11-16 07:15:46 UTC
Yeah, my commit is probably not going to help with your deadlock issue. Could you log another bug for that deadlock and assign it to Rodrigo? Thanks.