Bug 52437

Summary: Random NullReferenceExceptions in StringBuilder.ThreadSafeCopy
Product: [Mono] Runtime Reporter: Rolf Bjarne Kvinge [MSFT] <rolf>
Component: GCAssignee: Vlad Brezae <vlad.brezae>
Status: VERIFIED FIXED    
Severity: normal CC: agonzalez, kumpera, luis.aguilera, masafa, mono-bugs+mono, mono-bugs+runtime, vargaz
Priority: ---    
Version: 4.8.0 (C9)   
Target Milestone: C9SR0   
Hardware: PC   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: test case

Description Rolf Bjarne Kvinge [MSFT] 2017-02-10 10:24:01 UTC
I've seen random NullReferenceExceptions in StringBuilder.ThreadSafeCopy:

> System.NullReferenceException: Object reference not set to an instance of an object
>   at System.Text.StringBuilder.ThreadSafeCopy (System.Char* sourcePtr, System.Char[] destination, System.Int32 destinationIndex, System.Int32 count) [0x00007] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:1855 
>   at System.Text.StringBuilder.Append (System.Char* value, System.Int32 valueCount) [0x00095] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:1697 
>   at System.Text.StringBuilder.Append (System.Char[] value, System.Int32 startIndex, System.Int32 charCount) [0x00087] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:628 
>   at System.IO.StreamReader.ReadToEnd () [0x00029] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/io/streamreader.cs:449 
>   at xharness.RunDeviceTask.get_ProgressMessage () [0x00052] in /work/maccore/framework-sdk/xamarin-macios/tests/xharness/Jenkins.cs:2476 

> System.NullReferenceException: Object reference not set to an instance of an object
>   at System.Text.StringBuilder.ThreadSafeCopy (System.Char* sourcePtr, System.Char[] destination, System.Int32 destinationIndex, System.Int32 count) [0x00007] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:1855 
>   at System.Text.StringBuilder.Append (System.Char* value, System.Int32 valueCount) [0x00095] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:1697 
>   at System.Text.StringBuilder.Append (System.Char[] value, System.Int32 startIndex, System.Int32 charCount) [0x00087] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/text/stringbuilder.cs:628 
>   at System.Diagnostics.AsyncStreamReader.ReadBuffer (System.IAsyncResult ar) [0x0012f] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/System/services/monitoring/system/diagnosticts/AsyncStreamReader.cs:232 
>   at System.IO.Stream+ReadWriteTask.InvokeAsyncCallback (System.Object completedTask) [0x00015] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/io/stream.cs:670 
>   at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:957 
>   at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/executioncontext.cs:904 
>   at System.IO.Stream+ReadWriteTask.System.Threading.Tasks.ITaskCompletionAction.Invoke (System.Threading.Tasks.Task completingTask) [0x00060] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/io/stream.cs:696 
>   at System.Threading.Tasks.Task.FinishContinuations () [0x000a0] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:3635 
>   at System.Threading.Tasks.Task.FinishStageThree () [0x00045] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2366 
>   at System.Threading.Tasks.Task.FinishStageTwo () [0x000f8] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2339 
>   at System.Threading.Tasks.Task.Finish (System.Boolean bUserDelegateExecuted) [0x00049] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2239 
>   at System.Threading.Tasks.Task.ExecuteWithThreadLocal (System.Threading.Tasks.Task& currentTaskSlot) [0x00079] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2834 
>   at System.Threading.Tasks.Task.ExecuteEntry (System.Boolean bPreventDoubleExecution) [0x0006f] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2760 
>   at System.Threading.Tasks.Task.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/Tasks/Task.cs:2707 
>   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:854 
>   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /private/tmp/source-mono-4.8.0/bockbuild-mono-4.8.0-branch/profiles/mono-mac-xamarin/build-root/mono-x86/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1209 

Unfortunately I don't have a simple repro for it though, but I can probably debug this locally if someone have any idea where to start looking.

--------------------------------------------------------------------------------------------------------

$ mono --version
Mono JIT compiler version 4.8.0 (mono-4.8.0-branch/ba7f169 Fri Feb  3 13:34:24 EST 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  x86
	Disabled:      none
	Misc:          softdebug 
	LLVM:          yes(3.6.0svn-mono-master/8b1520c)
	GC:            sgen
Comment 1 Zoltan Varga 2017-02-10 18:30:21 UTC
Running with MONO_DEBUG=suspend-on-exception or suspend-on-unhandled will cause the process to busy loop when an exception happens so you can attach with lldb.
Comment 2 Rodrigo Kumpera 2017-02-13 21:40:08 UTC
This looks like a race condition in the BCL.
Comment 3 Marek Safar 2017-02-13 21:59:21 UTC
Did you see the code? There is not really that much code (and this is .net implementation) maybe there is some extra GC trick we need on Mono or iOS memcpy is not thread safe
Comment 4 Zoltan Varga 2017-02-13 22:36:58 UTC
Is it possible that 'sb' is null at that point ?
Comment 5 Rodrigo Kumpera 2017-02-14 00:09:03 UTC
Mono requires none of the GC tricks that dotnet requires, quite the opposite.

memcpy only matters when copying pointers and not a char[] and this is no x86 second to Rolf's report.
Comment 6 Marek Safar 2017-02-14 08:35:41 UTC
The call comes from StringBuilder instance method so that's quite unlikely
Comment 7 Rolf Bjarne Kvinge [MSFT] 2017-02-17 14:10:40 UTC
I tried to compile mono 4.8.0 locally with -O0, I couldn't reproduce (autogen.sh --prefix=/work/mono/4.8.0/install --enable-nls=no --enable-debug CFLAGS=-O0)
I compiled mono again, without -O0, still couldn't reproduce (autogen.sh --prefix=/work/mono/4.8.0/install --enable-nls=no --enable-debug)

My local mono:

> /work/mono/4.8.0/install/bin/mono --version
Mono JIT compiler version 4.8.0 (mono-4.8.0-branch/9ac5bf2 Mon Feb 13 08:27:04 CET 2017)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           normal
	SIGSEGV:       altstack
	Notification:  kqueue
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

So I tried with mono from a package again (the one from the initial description), and now I can reproduce again.

lldb session (I put a breakpoint on NullReferenceException in XS): https://gist.github.com/rolfbjarne/d4d5d119721eef7b4d4e322bad2a57dc

I've also been able to reproduce this without a debugger attached previously, so it's not the debugger.
Comment 8 Rolf Bjarne Kvinge [MSFT] 2017-02-17 15:55:04 UTC
Created attachment 19865 [details]
test case

This is some sort of OOM.

The following reproduces it (using the attached test case):

dd if=/dev/zero ibs=1k count=300000 | tr "\000" "\012" > foo.txt
mcs test.cs
mono test.exe foo.txt

sometimes it crashes with an assert somewhere, but most of the time this happens:

$ mono test.exe foo.txt 
Unhandled Exception:
System.NullReferenceException: Object reference not set to an instance of an object
  at System.Text.StringBuilder.ThreadSafeCopy (System.Char* sourcePtr, System.Char[] destination, System.Int32 destinationIndex, System.Int32 count) [0x00007] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Text.StringBuilder.Append (System.Char* value, System.Int32 valueCount) [0x00095] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Text.StringBuilder.Append (System.Char[] value, System.Int32 startIndex, System.Int32 charCount) [0x00087] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.IO.StreamReader.ReadToEnd () [0x00029] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at TestClass+<Main>c__AnonStorey0.<>m__0 () [0x0000e] in <b370cbca65124274aa863eb14f23da6f>:0 
  at System.Threading.ThreadHelper.ThreadStart_Context (System.Object state) [0x00017] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x0008d] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, System.Boolean preserveSyncCtx) [0x00000] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state) [0x00031] in <12e050e5b3d34326a1b4e2e7624e75da>:0 
  at System.Threading.ThreadHelper.ThreadStart () [0x0000b] in <12e050e5b3d34326a1b4e2e7624e75da>:0
Comment 9 Rodrigo Kumpera 2017-02-17 19:40:14 UTC
Hi Vlad,

Looks like mono is not throwing an OOM as it must.

To make this repro trivially, pass MONO_GC_PARAMS=max-heap-size=100m
Comment 10 Vlad Brezae 2017-02-28 13:43:43 UTC
*** Bug 51677 has been marked as a duplicate of this bug. ***
Comment 11 Vlad Brezae 2017-03-08 10:16:31 UTC
Fixed by f743f21ec746a942ee43a39915e405addc6f2501