Bug 23242

Summary: Null reference exception occurs after the call to Int.ToString() from multiple threads
Product: [Mono] Installers Reporter: Mike <mfilimonov>
Component: GeneralAssignee: Bugzilla <bugzilla>
Status: VERIFIED FIXED    
Severity: normal CC: adrian.murphy, kumpera, masafa, mohitk, mono-bugs+mono, mono-bugs+runtime, pj.beaman, ramc, rolf, vargaz
Priority: ---    
Version: unspecified   
Target Milestone: 3.12.0   
Hardware: PC   
OS: All   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: Source code for issue reproduction
Debug executable reproducing the issue

Description Mike 2014-09-22 09:09:34 UTC
The mono runtime is compiled from recent 'master' branch on github -i.e. Mono JIT compiler version 3.10
The system is
Linux 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
running on  Intel(R) Xeon(R) CPU E5-2680 with 32 hw threads support

In order to reproduce the issue you need to compile the attached source and execute it
/opt/mono/bin/mono-sgen --server ./getset.exe

In my case the app instantly crashes with the following stack trace:

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0 

Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Comment 1 Mike 2014-09-22 09:10:12 UTC
Created attachment 8143 [details]
Source code for issue reproduction
Comment 2 Mike 2014-09-22 10:05:11 UTC
Same issue on Mono JIT compiler version 3.10.1 (master/5f9c74f Mon Sep 22 14:03:21 UTC 2014)
(latest master from github)
running on 4-core vm
Linux ubuntu 3.13.0-32-generic #57-Ubuntu SMP Tue Jul 15 03:51:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
Comment 3 Mike 2014-09-22 10:26:38 UTC
Created attachment 8145 [details]
Debug executable reproducing the issue

Maybe the problem is related to IL code generation
This binary is build on
os x 10.10 (14A361p) (beta 3)
with mono runtime
Mono JIT compiler version 3.8.0 ((no/45d0ba1 Tue Aug 26 20:33:43 EDT 2014)
Comment 4 Rodrigo Kumpera 2014-09-22 10:49:36 UTC
Can you run it with --debug?
Comment 5 Mike 2014-09-22 11:17:02 UTC
Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:554 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00032] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:1019 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00026] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:945 
  at System.Int16.ToString () [0x00000] in /root/mono_off/mcs/class/corlib/System/Int16.cs:228 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00016] in /root/mono_off/mcs/class/corlib/System.Threading/Thread.cs:691 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:554 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00032] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:1019 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00026] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:945 
  at System.Int16.ToString () [0x00000] in /root/mono_off/mcs/class/corlib/System/Int16.cs:228 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00016] in /root/mono_off/mcs/class/corlib/System.Threading/Thread.cs:691 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:554 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00032] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:1019 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00026] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:945 
  at System.Int16.ToString () [0x00000] in /root/mono_off/mcs/class/corlib/System/Int16.cs:228 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00016] in /root/mono_off/mcs/class/corlib/System.Threading/Thread.cs:691 
[ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
  at System.NumberFormatter.ResetCharBuf (Int32 size) [0x00007] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:554 
  at System.NumberFormatter.FastIntegerToString (Int32 value, IFormatProvider fp) [0x00032] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:1019 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00026] in /root/mono_off/mcs/class/corlib/System/NumberFormatter.cs:945 
  at System.Int16.ToString () [0x00000] in /root/mono_off/mcs/class/corlib/System/Int16.cs:228 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00016] in /root/mono_off/mcs/class/corlib/System.Threading/Thread.cs:691
Comment 6 Mike 2014-09-22 11:17:39 UTC
Same issue while running with --debug on Mono JIT compiler version 3.8.0 ((detached/e451fb2 Mon Sep 22 15:09:03 UTC 2014)
Comment 7 Mike 2014-09-23 10:54:01 UTC
I've provided the requested info.
Comment 8 Mike 2014-09-23 12:32:23 UTC
Well, I've found a workaround for this issue - the mono runtime option --debug=mdb-optimizations
effectively 'disables' the crash.
Comment 9 Zoltan Varga 2014-09-23 19:15:41 UTC
Try running with -O=-aot, that might work around the problem.
Comment 10 Mike 2014-09-24 05:46:33 UTC
In that case a different exception occurs in one of 20 runs of getset.exe
 at System.Globalization.TextInfo..ctor (System.Globalization.CultureInfo ci, Int32 lcid, System.Void* data, Boolean read_only) [0x00000] in <filename unknown>:0 
  at System.Globalization.CultureInfo.CreateTextInfo (Boolean readOnly) [0x00000] in <filename unknown>:0 
  at System.Globalization.CultureInfo.ConstructInvariant (Boolean read_only) [0x00000] in <filename unknown>:0 

Unhandled Exception:
System.TypeInitializationException: An exception was thrown by the type initializer for System.Globalization.CultureInfo
  at System.Threading.Thread.get_CurrentCulture () [0x00000] in <filename unknown>:0 
  at System.NumberFormatter..ctor (System.Threading.Thread current) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.GetInstance (IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.NumberFormatter.NumberToString (Int32 value, IFormatProvider fp) [0x00000] in <filename unknown>:0 
  at System.Int16.ToString () [0x00000] in <filename unknown>:0 
  at getset.MainClass+<Main>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at System.Threading.Thread.StartInternal () [0x00000] in <filename unknown>:0
Comment 11 Mike 2014-09-24 07:37:50 UTC
I've reproduced the issue on OS X
Xamarin Studio Version 5.5 (build 198)
Runtime:
	Mono 3.10.0 ((detached/ac51002)
	GTK+ 2.24.23 (Raleigh theme)

Build Information
Release ID: 505000198
Git revision: 7495942eb76d6b80c460ddd61f2b94cba1a97fa2
Build date: 2014-09-18 09:50:12-04
Xamarin addins: c571b625445d60f2c8b189b309a6ffc87386caed

Operating System
Mac OS X 10.10.0 14.0.0 Darwin Kernel Version 14.0.0
    Mon Sep  8 05:27:41 PDT 2014
    root:xnu-2782.1.96~5/RELEASE_X86_64 x86_64
Comment 13 Marek Safar 2014-09-26 05:07:54 UTC
I can reproduce the issue too and as Zoltan suggested -O=-aot workarounds the issue
Comment 14 Mike 2014-09-26 16:28:58 UTC
1. It doesn't workaround the issue - if you run the executable in a loop - using the simple bash loop construct you will see that the exception still occurs even with -O=-aot
Well, it occurs for example in 31 iteration after 30 successful runs - but it still not a workaround.
The static constructor of EmptyArray class solves the problem 100%
2. AOT is really not an option for our real application because it uses emit
Comment 15 Marek Safar 2014-09-27 03:02:17 UTC
I ran it for 10 minutes without single error with -0=-aot on

ono JIT compiler version 3.10.0 (mono-3.10.0-branch/491d1f5 Wed 17 Sep 2014 11:23:27 CEST)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug
	LLVM:          supported, not enabled.
	GC:            sgen

static constructor for EmptyArray is wrong as we would have to fix all cases like this for any AOT-ed code including user code.
Comment 16 Zoltan Varga 2014-09-30 15:34:33 UTC
This is caused by the handling of got slots of type MONO_PATCH_INFO_SFLDA in the aot runtime.  Only the first thread which initializes the got slot waits for the type initializer to finish, the others don't.
Comment 17 Rolf Bjarne Kvinge [MSFT] 2014-11-17 10:32:22 UTC
I can repro this easily (50% of the time) with the initial test case.

> mono --version
Mono JIT compiler version 3.10.0 ((detached/633e444 Thu Oct  2 22:07:37 EDT 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug 
	LLVM:          yes(3.4svn-mono-(detached/e656cac)
	GC:            sgen

I can reproduce this with desktop mono, and there are users on the forums with released iOS apps running into it as well (http://forums.xamarin.com/discussion/comment/88543).
Comment 18 Zoltan Varga 2014-11-17 21:46:14 UTC
Fixed in mono master df8abf4920062fc93211ba2c1f65b77def5d9b1c and mono-3.12.0-branch f07e7d085010ef549df91611aaab029c244422d3. Thanks for the testcase.
Comment 21 Zoltan Varga 2014-12-09 07:57:09 UTC
It was a random failure, so it might not be reproducible to everyone. Doing 
mono --aot <path to mscorlib.dll> before running the test might make it more likely for the problem to occur.
Comment 22 Mohit Kheterpal 2014-12-09 08:21:17 UTC
As per comment 21, this issue is not easy to reproduce.

Hence, I am closing this issue by marking it as Verified.
Comment 23 Zoltan Varga 2015-01-07 11:13:29 UTC
*** Bug 4629 has been marked as a duplicate of this bug. ***