Bug 33191 - HttpListener race condition under load
Summary: HttpListener race condition under load
Status: RESOLVED NORESPONSE
Alias: None
Product: Class Libraries
Classification: Mono
Component: mscorlib (show other bugs)
Version: master
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Marek Safar
URL:
Depends on:
Blocks:
 
Reported: 2015-08-18 09:34 UTC by Christian Fahr
Modified: 2018-03-13 11:07 UTC (History)
5 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Source code (5.82 KB, text/x-csharp)
2015-08-18 09:35 UTC, Christian Fahr
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NORESPONSE

Description Christian Fahr 2015-08-18 09:34:29 UTC
HttpListener crashes under load with both exceptions in the managed threading code and native segmentation violations.
The code crashes during runtime (it will terminate after 10 seconds), if this would be fixed we still have to verify it does not crash during shutdown. (see https://bugzilla.xamarin.com/show_bug.cgi?id=32978)

To replicate:

mcs -debug Main.cs
mono --debug Main.exe

In another terminal:

ab2 -c1 -t11 "127.0.0.1:8088/" (for one request at a time)
or
ab2 -c10 -t11 "127.0.0.1:8088/" (for 10 concurrent requests)

You may need to try several times, but it crashes very lickely for me.

Used versions from git:

$ mcs --version
Mono C# compiler version 4.3.0.0
$ mono --version
Mono JIT compiler version 4.3.0 (master/79b2dea Tue Aug 18 10:52:04 CEST 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        LLVM:          supported, not enabled.
        GC:            sgen


Several exceptions:

System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00012] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/
itHandle.cs:198
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0002f] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.
ng/Timer.cs:338
  at System.Threading.Thread.StartInternal () [0x00016] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/T
s:696
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00012] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/
itHandle.cs:198
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0002f] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.
ng/Timer.cs:338
  at System.Threading.Thread.StartInternal () [0x00016] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/T
s:696

########################################################

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 295
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 366
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 295
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 366
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 10

########################################################

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) in <filename unknown>:line 0
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 369
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) in <filename unknown>:line 0
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 369
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105

########################################################

Stacktrace:


Native stacktrace:

        mono() [0x4a32c8]
        mono() [0x4f75be]
        mono() [0x424b62]
        /lib64/libpthread.so.0(+0xf890) [0x7f269608b890]
        mono() [0x49f3f6]
        mono() [0x4a2736]
        mono() [0x4f5cc9]
        /opt/mono/lib/mono/4.5/mscorlib.dll.so(+0x2eb74) [0x7f2693545b74]

Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
Comment 1 Christian Fahr 2015-08-18 09:35:21 UTC
Created attachment 12548 [details]
Source code
Comment 2 Christian Fahr 2015-08-18 09:39:28 UTC
Sorry, changed the port to 8080. Should have been ab2 -c10 -t11 "127.0.0.1:8080/".
Comment 3 Rodrigo Kumpera 2015-08-24 16:19:02 UTC
With master this doesn't hang in the ThreadPool but on AutoResetEvent + WaitAny.
Comment 4 Ludovic Henry 2015-08-31 12:09:04 UTC
I tested with master (https://github.com/mono/mono/commit/385c7692be38f92f91151d0c998a551d42b081f7) and it does not hang or crash, with 100 concurrent clients and for 100 seconds.

I am not seeing the hang in AutoResetEvent + WaitAny either, so marking this bug as resolved.
Comment 5 Christian Fahr 2015-09-02 10:16:10 UTC
Hi,


385c7 seems to be much more stable indeed. After several runs I still got both 
a native segmentation fault and

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) <0x7f373148fa40 + 0x00022> in <filename unknown>:0
  at System.Threading.Timer+Scheduler.SchedulerThread () <0x7f373148efa0 + 0x0076f> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x7f37312b9b50 + 0x0009a> in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906d0 + 0x001c6> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906a0 + 0x00020> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x7f37312905f0 + 0x00059> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () <0x7f37312b9d10 + 0x0002e> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) <0x7f373148fa40 + 0x00022> in <filename unknown>:0
  at System.Threading.Timer+Scheduler.SchedulerThread () <0x7f373148efa0 + 0x0076f> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x7f37312b9b50 + 0x0009a> in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906d0 + 0x001c6> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906a0 + 0x00020> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x7f37312905f0 + 0x00059> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () <0x7f37312b9d10 + 0x0002e> in <filename unknown>:0


With the current master/39cbd2e its much more unstable again:

Native stacktrace:

        mono() [0x4a2f78]
        mono() [0x4f70de]
        mono() [0x424c12]
        /lib64/libpthread.so.0(+0xf890) [0x7fba00731890]
        mono() [0x49f3d6]
        mono() [0x4a2510]
        mono() [0x4f57e9]
        /opt/mono-master/lib/mono/4.5/mscorlib.dll.so(+0x2eb84) [0x7fb9fd945b84]

Can someone confirm?
Comment 6 Rodrigo Kumpera 2015-09-02 13:53:06 UTC
Maybe the crash is linux only. I could not reproduce on mac.
Comment 7 Rolf Bjarne Kvinge [MSFT] 2015-11-04 04:49:54 UTC
This might be related to bug #29299 and bug #29212, which only happens on certain linux kernels.

@Christian, which OS (and version) are you running?
Comment 8 Marek Safar 2018-03-13 11:07:13 UTC
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and reopen the bug report.

Thank you!