Bug 41914

Summary: Race condition in named mutex
Product: [Mono] Runtime Reporter: Mikhail Filippov <mikhail>
Component: io-layerAssignee: Ludovic Henry <ludovic>
Status: RESOLVED FIXED    
Severity: normal CC: awiouy, kumpera, ludovic, mono-bugs+mono, mono-bugs+runtime
Priority: Normal    
Version: 4.5.X   
Target Milestone: Future Cycle (TBD)   
Hardware: PC   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:

Description Mikhail Filippov 2016-06-16 21:59:30 UTC
I found problem in Mono in named mutex I have repro program:
using System;
using System.Threading;

namespace Crasher
{
        internal class Program
        {
                public static void Main (string[] args)
                {
                        var a = "";
                        for (var i = 0; i < 100; i++)
                        {				
                                new Thread(Crasher).Start();
                        }
                        Console.WriteLine(a);
                        Console.ReadLine();
                }

                private static void Crasher(){
                        var rnd = new Random();
                        while (true)
                        {
                                Thread.Sleep(rnd.Next(100, 10000));
                                using (var mutex = new Mutex(false, "Global\\TEST"))
                                {
                                        var owner = false;
                                        try
                                        {
                                                owner = mutex.WaitOne(TimeSpan.FromMinutes(1));
                                        }
                                        finally
                                        {
                                                if (owner)
                                                {
                                                        mutex.ReleaseMutex();
                                                }
                                        }
                                        Console.WriteLine ("PING");
                                }
                                Thread.Sleep(rnd.Next(100, 10000));
                        }
                }
        }
}

It program crached mono after few seconds.
I see namedmutex_create and namedmutex_release use different locks.
First time I found it problem when run Nuget client on mono.
I verify repro on mono 4.2, 4.4, and master.
Anybody can help me resolve this issue?
On 4.4.0 native crash:
namedmutex_create: error creating mutex handle
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
_wapi_handle_unref_full: Attempting to unref unused handle 0x4e0
PING
PING
namedmutex_create: error creating mutex handle
PING
PING
PING
PING
PING
PING
_wapi_handle_ref: Attempting to ref unused handle 0x4e3
* Assertion at ../../mono/utils/mono-os-mutex.h:135, condition `res != EINVAL' not met

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.WaitHandle.WaitOne_internal (System.Threading.WaitHandle,intptr,int,bool) <0x00073>
  at System.Threading.WaitHandle.WaitOne (System.TimeSpan,bool) <0x0009b>
  at System.Threading.WaitHandle.WaitOne (System.TimeSpan) <0x0001d>
  at Crasher.Program.Crasher () <0x000f0>
  at System.Threading.ThreadHelper.ThreadStart_Context (object) <0x0009a>
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x001c6>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object,bool) <0x00020>
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext,System.Threading.ContextCallback,object) <0x00059>
  at System.Threading.ThreadHelper.ThreadStart () <0x0002e>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <0x000e0>

Native stacktrace:

        0   mono                                0x0000000103f9b0ca mono_handle_native_sigsegv + 271
        1   libsystem_platform.dylib            0x00007fff854c252a _sigtramp + 26
        2   mono                                0x00000001042036a4 tmp_dir + 5316
        3   libsystem_c.dylib                   0x00007fff935536e7 abort + 129
        4   mono                                0x000000010410d1f0 monoeg_g_log + 0
        5   mono                                0x000000010410d175 monoeg_g_logv + 83
        6   mono                                0x000000010410d31a monoeg_assertion_message + 143
        7   mono                                0x00000001040e453d _wapi_handle_timedwait_signal_handle + 1153
        8   mono                                0x00000001040f4aec wapi_WaitForSingleObjectEx + 606
        9   mono                                0x000000010406c01c mono_wait_uninterrupted + 130
        10  mono                                0x000000010406c1ff ves_icall_System_Threading_WaitHandle_WaitOne_internal + 73
        11  ???                                 0x0000000108130b54 0x0 + 4430433108
        12  mscorlib.dll.dylib                  0x00000001062ed7ae System_Threading_WaitHandle_WaitOne_System_TimeSpan + 30
        13  mscorlib.dll.dylib                  0x0000000106140e5b System_Threading_ThreadHelper_ThreadStart_Context_object + 155
        14  mscorlib.dll.dylib                  0x000000010613f331 System_Threading_ExecutionContext_Run_System_Threading_ExecutionContext_System_Threading_ContextCallback_object_bool + 33
        15  mono                                0x0000000103f04876 mono_jit_runtime_invoke + 1578
        16  mono                                0x0000000104090c23 mono_runtime_invoke + 130
        17  mono                                0x0000000104070409 start_wrapper + 424
        18  mono                                0x0000000104106cb1 inner_start_thread + 305
        19  libsystem_pthread.dylib             0x00007fff989c399d _pthread_body + 131
        20  libsystem_pthread.dylib             0x00007fff989c391a _pthread_body + 0
        21  libsystem_pthread.dylib             0x00007fff989c1351 thread_start + 13



On master UnhandlerException:
namedmutex_create: error creating mutex handle
PING
PING
PING
PING
PING
_wapi_handle_unref_full: Attempting to unref unused handle 0x4eb
PING
PING
PING
_wapi_handle_unref_full: Attempting to unref unused handle 0x4ec
PING
PING
PING
PING
PING
PING
PING
PING
_wapi_handle_unref_full: Attempting to unref unused handle 0x4ee
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
PING
_wapi_handle_unref_full: Attempting to unref unused handle 0x4f1
PING
PING
PING
PING
PING
PING
PING

Unhandled Exception:
System.ApplicationException: Mutex is not owned
  at System.Threading.Mutex.ReleaseMutex () <0x10e769ef0 + 0x00043> in <filename unknown>:0 
  at (wrapper remoting-invoke-with-check) System.Threading.Mutex:ReleaseMutex ()
  at Crasher.Program.Crasher () <0x10cfdfc80 + 0x00129> in <filename unknown>:0 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) <0x10e5fb710 + 0x00096> in <filename unknown>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) <0x10e5f93b0 + 0x001c6> in <filename unknown>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) <0x10e5f9380 + 0x00020> in <filename unknown>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) <0x10e5f92d0 + 0x00059> in <filename unknown>:0 
  at System.Threading.ThreadHelper.ThreadStart () <0x10e5fb8c0 + 0x0002e> in <filename unknown>:0 
_wapi_handle_unref_full: Attempting to unref unused handle 0x4f1
[ERROR] FATAL UNHANDLED EXCEPTION: Nested exception trying to figure out what went wrong
Comment 1 Rodrigo Kumpera 2016-09-14 01:41:33 UTC
Still can repro with master.

I think the issue is that mono_w32handle_namespace_search_handle should ref before returning.

ref'ing outside of the scan_lock is buggy by definition.
Comment 2 Ludovic Henry 2016-09-14 15:57:15 UTC
@kumpera I agree with you, I can confirm that this is the issue. By trying to fix it, I am also going down the rabbit hole of handle lifetime management when looping over them.

@mikhail actively working on it, coming back to you as soon as I have something new.
Comment 3 Ludovic Henry 2016-09-14 17:17:30 UTC
This is fixed with https://github.com/mono/mono/pull/3560
Comment 4 Anton Voyl 2017-01-05 08:10:15 UTC
I encounter similar symptoms when running Emby with latest Mono 4.6 for arm.
Will the fix be included in Mono 4.6?