Bug 21623 - Frequent crashes (SIGABRT) on application exit when using System.Diagnostics.Process
Summary: Frequent crashes (SIGABRT) on application exit when using System.Diagnostics....
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: GC (show other bugs)
Version: 3.4.0
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2014-07-27 04:44 UTC by delcypher
Modified: 2017-10-11 17:08 UTC (History)
8 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Program that frequently crashes under mono 3.4 (984 bytes, text/x-csharp)
2014-07-27 04:44 UTC, delcypher
Details
Crash output when running program from console (4.32 KB, text/plain)
2014-07-27 04:45 UTC, delcypher
Details
Crash output when running program from monodevelop (5.93 KB, text/plain)
2014-07-27 04:45 UTC, delcypher
Details
Output of mono crashing with DEBUG macros in ``mono/io-layer/handles.c`` enabled. (17.67 KB, text/plain)
2014-07-29 12:11 UTC, delcypher
Details
gdb (12.70 KB, text/plain)
2014-09-08 13:06 UTC, xamarin
Details
test-program (395 bytes, text/plain)
2014-09-08 13:06 UTC, xamarin
Details
Compiler segfault introduced in a0afa38296b8a3b0382bf34ce777357d2553c0f0 (15.47 KB, text/plain)
2014-09-08 13:36 UTC, delcypher
Details
Task example to reproduce the issue (661 bytes, text/plain)
2015-10-15 09:39 UTC, Peter Beeby
Details
debug and trace output from Task code sample (149.05 KB, text/plain)
2015-10-16 05:42 UTC, Peter Beeby
Details

Description delcypher 2014-07-27 04:44:53 UTC
Created attachment 7498 [details]
Program that frequently crashes under mono 3.4

I recently upgraded from Mono 3.2 to 3.4 and I noticed that my application was frequently crashing on exit.

I've reduced it down to a simple program that launches the sleep command for a few seconds. When the program tries to terminate the runtime frequently crashes with SIGABRT. Crashes do not occur every time they usually start with

```
Error destroying handle 0x40d mutex due to 16

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.InternalThread.Thread_free_internal (System.Threading.InternalThread,intptr) <IL 0x0001a, 0xffffffff>
  at System.Threading.InternalThread.Finalize () [0x00000] in /home/dan/dev/aur/mono/src/mono-3.4.0/mcs/class/corlib/System.Threading/Thread.cs:117
  at (wrapper runtime-invoke) object.runtime_invoke_virtual_void__this__ (object,intptr,intptr,intptr) <IL 0x0004c, 0xffffffff>
```

I have attached the test program and output produced when crashes occur.
Comment 1 delcypher 2014-07-27 04:45:27 UTC
Created attachment 7499 [details]
Crash output when running program from console
Comment 2 delcypher 2014-07-27 04:45:51 UTC
Created attachment 7500 [details]
Crash output when running program from monodevelop
Comment 3 delcypher 2014-07-27 04:51:11 UTC
This may be related to

https://bugzilla.xamarin.com/show_bug.cgi?id=19140

It is also a little similar to
https://bugzilla.xamarin.com/show_bug.cgi?id=17894

I should of also stated that

- I do not observe the issue with mono3.2
- I'm using Arch Linux and it has been patched slightly ( see https://projects.archlinux.org/svntogit/packages.git/tree/trunk?h=packages/mono )
Comment 4 Rodrigo Kumpera 2014-07-27 22:56:31 UTC
I can't repro the crash with your test case on OSX and mono master.

Does it still happen to you?
Comment 5 delcypher 2014-07-28 05:06:29 UTC
I still have the issue but I have not tried using mono from master. I'll try this  during the week and will report back.
Comment 6 delcypher 2014-07-29 12:09:53 UTC
I just tried compiling from master and I still have the same issue. In fact I had to downgrade to mono-3.2 to compile mono in master because using mono3.4 the compilation process kept on crashing with the same ``Error destroying handle...`` message.

It seems the issue is easier to manifest if I do this

```
for ((i=0; i<30 ; ++i)) ; do (mono test/CrashTest/CrashTest/bin/Debug/CrashTest.exe &) ; done
```

which will launch mono many times in parallel.

I see the bug in at least a couple of the runs which indicates that this is some sort of race condition. I have another Linux machine with a near identical install using mono3.4 and I cannot reproduce this issue :(

I took a brief look at the code. The call in mono/io-layer/handles.c

```
g_error ("Error destroying handle %p mutex due to %d\n", handle, thr_ret);
```

triggers the abort.

A comment near by noting "/*WARNING gross hack to make cleanup not crash when exiting without the whole runtime teardown.*/" does not fill me with confidence...


I noticed there were a bunch of macros lying around

```
#undef DEBUG_REFS

#if 1
#define DEBUG(...) g_message(__VA_ARGS__)
```

I changed these to

```
#define DEBUG_REFS

#if 1
#define DEBUG(...) g_message(__VA_ARGS__)
```
and also had to change

```
DEBUG ("%s: unreffing %s handle %p", __func__, _wapi_handle_typename[type], handle);
```

to

```
DEBUG ("%s: unreffing XXX handle %p", __func__, handle);
```

because the variable ``type`` doesn't exist which is used to index into _wapi_handle_typename.

I've uploaded the output of a crashing invocation of mono with these extra debug messages in the hope that it will be useful.

$ mono CrashTest.exe 2>&1 | tee mono-crash-with-debug-macros.txt
Comment 7 delcypher 2014-07-29 12:11:27 UTC
Created attachment 7518 [details]
Output of mono crashing with DEBUG macros in ``mono/io-layer/handles.c`` enabled.

Output of mono crashing with DEBUG macros in ``mono/io-layer/handles.c`` enabled.
Comment 8 João Matos 2014-07-31 07:34:32 UTC
I can't repro this crasher either with 64-bit Ubuntu.

I tried both master and the version that comes with Ubuntu.

Mono JIT compiler version 3.8.1 (master/13f4ba5 Thu Jul 31 13:06:30 CEST 2014)

Mono JIT compiler version 3.2.8 (Debian 3.2.8+dfsg-4ubuntu1)
Comment 9 delcypher 2014-08-06 13:51:24 UTC
Thanks for trying to reproduce the issue.

I think the issue must be a combination of my hardware and software because

- I can't reproduce the issue on a different machine (different hardware) running exactly the same software as the machine that shows the issue.

- A colleague of mine has the same hardware as the machine that shows the issue but is running Ubuntu 14.04LTS. He couldn't reproduce the issue either.


How should I proceed? The issue isn't present in mono3.2 so I could do a git bisect until I find the offending commit but this could be a huge waste of my time if no one else can produce the issue (and therefore probably wouldn't accept reverting a commit).
Comment 10 João Matos 2014-08-06 14:52:49 UTC
If you could find the offending commit via bisect that'd be awesome. That'd make it much easier to figure out what is going on...
Comment 11 xamarin 2014-09-08 13:06:09 UTC
Created attachment 7963 [details]
gdb
Comment 12 xamarin 2014-09-08 13:06:35 UTC
Created attachment 7964 [details]
test-program
Comment 13 xamarin 2014-09-08 13:07:08 UTC
I'm running into same problem in (all?) multi threaded programs, I've attached test program and log from gdb.
Comment 14 delcypher 2014-09-08 13:28:57 UTC
Well I'm glad someone else can reproduce this issue.

I actually started trying to do a git bisect on mono yesterday and I ran into a lot of trouble.

- Mono's git history is non-linear so I can't do a bisect across branches. I worked around this by using ``git merge-base`` to find a commit I could use on the master branch and started the bisect from there.

- There are commits in the history that don't compile!

I hit one commit that I hit whilst trying bisect which is a little suspicious

```
commit a0afa38296b8a3b0382bf34ce777357d2553c0f0
Author: Zoltan Varga <vargaz@gmail.com>
Date:   Mon Jan 13 04:00:58 2014 +0100

    Move the mono_thread_create () function into utils/mono-threads.h/c, change/simplify its signature a bit.
```

Before this commit I can compile mono without errors and I don't experience any of my mono program crashing.

If I try building mono from this commit the bootstrap compiler segfaults in libpthread (I think, see attached log) so I can't even build mono. This commit however might however been on unrelated to the bug because I know at some point in the future commits I am able to compile mono again but I get crashes in my test program. I didn't know about the ``git bisect skip`` command when I did this yesterday so I might try to do the bisect again and skip this commit to see if I can correctly pinpoint the commit that introduces the bug.
Comment 15 delcypher 2014-09-08 13:36:18 UTC
Created attachment 7965 [details]
Compiler segfault introduced in a0afa38296b8a3b0382bf34ce777357d2553c0f0
Comment 16 xamarin 2014-10-05 06:34:15 UTC
This bug still happens in 3.10.

I've rolled back to 3.2.8 (debian-testing), and it does NOT crash there.
Comment 17 obadz 2014-11-29 18:36:16 UTC
Same bug here using Mono 3.10 from http://download.mono-project.com/repo/debian on Ubuntu 14.10

$ nuget install Newtonsoft.Json
Installing 'Newtonsoft.Json 6.0.6'.
Successfully installed 'Newtonsoft.Json 6.0.6'.
Error destroying handle 0x40c mutex due to 16

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.InternalThread.Thread_free_internal (System.Threading.InternalThread,intptr) <0xffffffff>
  at System.Threading.InternalThread.Finalize () <0x0001b>
  at (wrapper runtime-invoke) object.runtime_invoke_virtual_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

	/usr/bin/cli() [0x4b3f7c]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0xfc90) [0x7f13d1ddac90]
	/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37) [0x7f13d1a3cd27]
	/lib/x86_64-linux-gnu/libc.so.6(abort+0x148) [0x7f13d1a3e418]
	/usr/bin/cli() [0x636e59]
	/usr/bin/cli() [0x637067]
	/usr/bin/cli() [0x637112]
	/usr/bin/cli() [0x609d78]
	/usr/bin/cli() [0x60b3c4]
	/usr/bin/cli() [0x58511e]
	[0x40499451]
Comment 18 Zoltan Varga 2014-11-29 20:42:54 UTC
This will be fixed in the upcoming mono 3.12.

*** This bug has been marked as a duplicate of bug 23423 ***
Comment 19 Zoltan Varga 2015-01-16 13:15:35 UTC
It looks like this problem might be different than #23423, reopening.
Comment 20 Peter Beeby 2015-10-15 09:39:28 UTC
Created attachment 13343 [details]
Task example to reproduce the issue
Comment 21 Peter Beeby 2015-10-15 09:41:55 UTC
We have been able to reproduce this issue. We use a custom build of openSUSE 13.1 booting from USB drive and have experienced this issue under mono versions 4.0.3 and 4.0.4, I also tried updating to the alpha 4.2 and this also produces the issue.

The issue seems slightly harder to achieve using the alpha but I discovered the issue using a different code example (above), I could replicate the issue with only 2 tasks up to the alpha, after that I needed approx 500

What I have discovered is it seems to be specific to certain hardware. I have used a boot-able thumb drive with the OS on many different hardware platforms.
the hardware ranges across the i3-i7 processor types and some AMD systems.

Of the 20-30 systems I have tested I have reproduced the issue on 3 systems, the processors in these systems are: 
i5-4690, Gigabyte GA-B85M-DS3H, 16GB Kingston Hyper Beast Ram
i7-4790k, AsRock Z87 Extreme4/TB4, 16GB Kingston Fury Hyperx
The third system is a laptop: Toshiba Portege z30 with an i5 4300U 1.9ghz

the resulting error:

Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Threading.NativeEventCalls.CloseEvent_internal (intptr) <0xffffffff>
  at Microsoft.Win32.SafeHandles.SafeWaitHandle.ReleaseHandle () <0x00013>
  at System.Runtime.InteropServices.SafeHandle.Dispose (bool) <0x00065>
  at System.Runtime.InteropServices.SafeHandle.Finalize () <0x0001a>
  at (wrapper runtime-invoke) object.runtime_invoke_virtual_void__this__ (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

	mono-boehm() [0x4b9532]
	/lib64/libpthread.so.0(+0xf9f0) [0x7fc4da19d9f0]
	/lib64/libc.so.6(gsignal+0x39) [0x7fc4d9bfd849]
	/lib64/libc.so.6(abort+0x148) [0x7fc4d9bfecd8]
	mono-boehm() [0x61c539]
	mono-boehm() [0x61c7ac]
	mono-boehm() [0x61c87f]
	mono-boehm() [0x5dd0f8]
	mono-boehm() [0x5deaa4]
	[0x41d1ccf2]
Comment 22 Rodrigo Kumpera 2015-10-15 12:58:06 UTC
Can you install symbols and provide a backtrace with proper function names and line information?
Comment 23 João Matos 2015-10-15 20:07:32 UTC
Ran your test case on loop on OSX for a couple hours and was not able to reproduce any crash.
Comment 24 Peter Beeby 2015-10-16 05:42:59 UTC
Created attachment 13366 [details]
debug and trace output from Task code sample

I have attached the output from running the sample I uploaded, is this what you require?

When running the code under debug and trace I found that it didn't fail every time, this might suggest its a timing issue?
Comment 25 Rodrigo Kumpera 2015-10-16 11:23:46 UTC
João, this might be linux specific.
Comment 26 Peter Beeby 2015-10-19 07:34:38 UTC
This issue seems very dependent on the hardware too, we have switched the i5-4690 processor to a i5-4440 and the issue goes away
Comment 27 Rodrigo Kumpera 2017-10-11 17:08:10 UTC
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and reopen the bug report.

Thank you!

Note You need to log in before you can comment on or make changes to this bug.