Bug 33191 - HttpListener race condition under load
Summary: HttpListener race condition under load
Status: NEEDINFO
Alias: None
Product: Class Libraries
Classification: Mono
Component: mscorlib (show other bugs)
Version: master
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Marek Safar
URL:
Depends on:
Blocks:
 
Reported: 2015-08-18 09:34 UTC by Christian Fahr
Modified: 2015-11-04 04:49 UTC (History)
5 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Source code (5.82 KB, text/x-csharp)
2015-08-18 09:35 UTC, Christian Fahr
Details

Description Christian Fahr 2015-08-18 09:34:29 UTC
HttpListener crashes under load with both exceptions in the managed threading code and native segmentation violations.
The code crashes during runtime (it will terminate after 10 seconds), if this would be fixed we still have to verify it does not crash during shutdown. (see https://bugzilla.xamarin.com/show_bug.cgi?id=32978)

To replicate:

mcs -debug Main.cs
mono --debug Main.exe

In another terminal:

ab2 -c1 -t11 "127.0.0.1:8088/" (for one request at a time)
or
ab2 -c10 -t11 "127.0.0.1:8088/" (for 10 concurrent requests)

You may need to try several times, but it crashes very lickely for me.

Used versions from git:

$ mcs --version
Mono C# compiler version 4.3.0.0
$ mono --version
Mono JIT compiler version 4.3.0 (master/79b2dea Tue Aug 18 10:52:04 CEST 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug 
        LLVM:          supported, not enabled.
        GC:            sgen


Several exceptions:

System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00012] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/
itHandle.cs:198
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0002f] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.
ng/Timer.cs:338
  at System.Threading.Thread.StartInternal () [0x00016] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/T
s:696
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.EventWaitHandle.Reset () [0x00012] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/
itHandle.cs:198
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Reset ()
  at System.Threading.Timer+Scheduler.SchedulerThread () [0x0002f] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.
ng/Timer.cs:338
  at System.Threading.Thread.StartInternal () [0x00016] in /home/abuild/rpmbuild/BUILD/mono-4.0.2/mcs/class/corlib/System.Threading/T
s:696

########################################################

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 295
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 366
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.Add (System.Threading.Timer timer) in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 295
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 366
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 10

########################################################

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) in <filename unknown>:line 0
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 369
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
   at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) in <filename unknown>:line 0
   at System.Threading.Timer+Scheduler.SchedulerThread () in /root/mono/mcs/class/corlib/System.Threading/Timer.cs:line 369
   at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 68
   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 581
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 530
   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) in /root/mono/external/referencesource/mscorlib/system/threading/executioncontext.cs:line 519
   at System.Threading.ThreadHelper.ThreadStart () in /root/mono/external/referencesource/mscorlib/system/threading/thread.cs:line 105

########################################################

Stacktrace:


Native stacktrace:

        mono() [0x4a32c8]
        mono() [0x4f75be]
        mono() [0x424b62]
        /lib64/libpthread.so.0(+0xf890) [0x7f269608b890]
        mono() [0x49f3f6]
        mono() [0x4a2736]
        mono() [0x4f5cc9]
        /opt/mono/lib/mono/4.5/mscorlib.dll.so(+0x2eb74) [0x7f2693545b74]

Debug info from gdb:


=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================
Comment 1 Christian Fahr 2015-08-18 09:35:21 UTC
Created attachment 12548 [details]
Source code
Comment 2 Christian Fahr 2015-08-18 09:39:28 UTC
Sorry, changed the port to 8080. Should have been ab2 -c10 -t11 "127.0.0.1:8080/".
Comment 3 Rodrigo Kumpera 2015-08-24 16:19:02 UTC
With master this doesn't hang in the ThreadPool but on AutoResetEvent + WaitAny.
Comment 4 Ludovic Henry 2015-08-31 12:09:04 UTC
I tested with master (https://github.com/mono/mono/commit/385c7692be38f92f91151d0c998a551d42b081f7) and it does not hang or crash, with 100 concurrent clients and for 100 seconds.

I am not seeing the hang in AutoResetEvent + WaitAny either, so marking this bug as resolved.
Comment 5 Christian Fahr 2015-09-02 10:16:10 UTC
Hi,


385c7 seems to be much more stable indeed. After several runs I still got both 
a native segmentation fault and

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) <0x7f373148fa40 + 0x00022> in <filename unknown>:0
  at System.Threading.Timer+Scheduler.SchedulerThread () <0x7f373148efa0 + 0x0076f> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x7f37312b9b50 + 0x0009a> in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906d0 + 0x001c6> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906a0 + 0x00020> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x7f37312905f0 + 0x00059> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () <0x7f37312b9d10 + 0x0002e> in <filename unknown>:0
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.Threading.Timer+Scheduler.ShrinkIfNeeded (System.Collections.Generic.List`1 list, Int32 initial) <0x7f373148fa40 + 0x00022> in <filename unknown>:0
  at System.Threading.Timer+Scheduler.SchedulerThread () <0x7f373148efa0 + 0x0076f> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x7f37312b9b50 + 0x0009a> in <filename unknown>:0
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906d0 + 0x001c6> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f37312906a0 + 0x00020> in <filename unknown>:0
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x7f37312905f0 + 0x00059> in <filename unknown>:0
  at System.Threading.ThreadHelper.ThreadStart () <0x7f37312b9d10 + 0x0002e> in <filename unknown>:0


With the current master/39cbd2e its much more unstable again:

Native stacktrace:

        mono() [0x4a2f78]
        mono() [0x4f70de]
        mono() [0x424c12]
        /lib64/libpthread.so.0(+0xf890) [0x7fba00731890]
        mono() [0x49f3d6]
        mono() [0x4a2510]
        mono() [0x4f57e9]
        /opt/mono-master/lib/mono/4.5/mscorlib.dll.so(+0x2eb84) [0x7fb9fd945b84]

Can someone confirm?
Comment 6 Rodrigo Kumpera 2015-09-02 13:53:06 UTC
Maybe the crash is linux only. I could not reproduce on mac.
Comment 7 Rolf Bjarne Kvinge [MSFT] 2015-11-04 04:49:54 UTC
This might be related to bug #29299 and bug #29212, which only happens on certain linux kernels.

@Christian, which OS (and version) are you running?

Note You need to log in before you can comment on or make changes to this bug.