Bug 18151 - SEG Faults
Summary: SEG Faults
Status: NEEDINFO
Alias: None
Product: Runtime
Classification: Mono
Component: GC (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2014-03-04 08:00 UTC by Greg Young
Modified: 2017-10-11 17:12 UTC (History)
9 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Greg Young 2014-03-04 08:00:28 UTC
We have been seeing seg faults in sgen. I am including some backtraces they have also been put on mono-dev list. This is in trunk running in ubuntu. We see the same issues whether statically linked + AOT or whether just running a standard process.

The most common (can usually get within a stress test within 2-20m iterations) is:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed7f2700 (LWP 3886)]
copy_object_no_checks (obj=obj@entry=0x7ffff6683150, 
    queue=queue@entry=0x9831c0 <gray_queue>) at sgen-copy-object.h:110
110		gboolean has_references = SGEN_VTABLE_HAS_REFERENCES (vt);
(gdb) backtrace
#0  copy_object_no_checks (obj=obj@entry=0x7ffff6683150, 
    queue=queue@entry=0x9831c0 <gray_queue>) at sgen-copy-object.h:110
#1  0x00000000005fdec3 in simple_nursery_serial_copy_object_from_obj (
    queue=0x9831c0 <gray_queue>, obj_slot=0x7fffcc8c09c0)
    at sgen-minor-copy-object.h:206
#2  simple_nursery_serial_scan_object (start=<optimized out>, 
    queue=0x9831c0 <gray_queue>) at sgen-scan-object.h:64
#3  0x00000000005d9aff in sgen_drain_gray_stack (max_objs=max_objs@entry=-1, 
    ctx=...) at sgen-gc.c:1194
#4  0x00000000005df36e in collect_nursery (unpin_queue=unpin_queue@entry=0x0, 
    finish_up_concurrent_mark=finish_up_concurrent_mark@entry=0)
    at sgen-gc.c:2638
#5  0x00000000005df839 in collect_nursery (finish_up_concurrent_mark=0, 
    unpin_queue=0x0) at sgen-gc.c:3554
#6  sgen_perform_collection (requested_size=4096, generation_to_collect=0, 
    reason=0x70b2e9 "Nursery full", wait_to_finish=0) at sgen-gc.c:3490
#7  0x00000000005f5dd9 in mono_gc_alloc_obj_nolock (
    vtable=vtable@entry=0xab8680, size=size@entry=576) at sgen-alloc.c:288
#8  0x00000000005f5fe3 in mono_gc_alloc_vector (vtable=0xab8680, size=576, 
    max_length=270) at sgen-alloc.c:499
#9  0x00000000400147f9 in ?? ()
#10 0x00007fffb40025d0 in ?? ()
#11 0x0000000000000000 in ?? ()




We also sometimes see though far rarer:
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffed5f1700 (LWP 16236)]
0x00000000005e1270 in alloc_obj (vtable=0x7ffff662e0e8, size=-1450344072, 
    pinned=0, has_references=1) at sgen-marksweep.c:740
740		int size_index = MS_BLOCK_OBJ_SIZE_INDEX (size);
(gdb) backtrace
#0  0x00000000005e1270 in alloc_obj (vtable=0x7ffff662e0e8, size=-1450344072, 
    pinned=0, has_references=1) at sgen-marksweep.c:740
#1  0x00000000005fb5f4 in alloc_for_promotion (has_references=1, 
    objsize=2844623224, obj=0x7ffff662df90 "\350\340b\366\377\177", 
    vtable=0x7ffff662e0e8) at sgen-simple-nursery.c:35
#2  copy_object_no_checks (obj=obj@entry=0x7ffff662df90, 
    queue=queue@entry=0x983120 <gray_queue>) at sgen-copy-object.h:112
#3  0x00000000005fc382 in simple_nursery_serial_copy_object_from_obj (
    queue=0x983120 <gray_queue>, obj_slot=0x7fffd5fac5b0)
    at sgen-minor-copy-object.h:206
#4  simple_nursery_serial_scan_object (start=<optimized out>, 
    queue=0x983120 <gray_queue>) at sgen-scan-object.h:64
#5  0x00000000005d8a6f in sgen_drain_gray_stack (max_objs=max_objs@entry=-1, 
    ctx=...) at sgen-gc.c:1194
#6  0x00000000005de27e in collect_nursery (unpin_queue=unpin_queue@entry=0x0, 
    finish_up_concurrent_mark=finish_up_concurrent_mark@entry=0)
    at sgen-gc.c:2631
#7  0x00000000005de749 in collect_nursery (finish_up_concurrent_mark=0, 
    unpin_queue=0x0) at sgen-gc.c:3547
#8  sgen_perform_collection (requested_size=4096, generation_to_collect=0, 
    reason=0x70b51a "Nursery full", wait_to_finish=0) at sgen-gc.c:3483
#9  0x00000000005f4b49 in mono_gc_alloc_obj_nolock (vtable=0x195aed8, size=32)
    at sgen-alloc.c:288
#10 0x00000000005f4c14 in mono_gc_alloc_obj (vtable=0x195aed8, size=32)
    at sgen-alloc.c:465
Comment 1 Paolo Molaro 2014-03-11 09:24:34 UTC
Do you have a test case you can share for this?
Comment 2 Greg Young 2014-03-17 14:03:10 UTC
ok so the test is a bit involved though Rodrigo has it.

You can either:

a) pull dev on github.com/eventstore/eventstore and build from source (there is a script to do this but is a bit involved)
b) grab these linux binaries http://ha.geteventstore.com/showcase/EventStore-Mono-v2.5.0rc4.tar.gz (mono statically linked with them): 

Once you have done either go to the binary directory. Run:

EventStore.Singlenode.exe --run-projections=all

in another terminal

EventStore.TestClient.exe

this is a command window. Type WRFL 10 50000000 it will fail within a few minutes with a seg fault.

All we are doing here is write flooding the database with 50m requests (eg a stress test)

Cheers,

Greg
Comment 3 Greg Young 2014-03-20 08:40:30 UTC
Any feedback on this? Is there anything we can do to help?
Comment 4 Miguel de Icaza [MSFT] 2014-04-15 21:38:54 UTC
Greg,

Please include the source code to this bug report, and flag it as a private attachment.

It is pointless for us to look at a binary that contains a statically linked version of Mono.  We have no idea what is in there, nor a way to test any changes, nor verify anything.

Setting the bug to NEEDINFO, until we get the source code attached to this bug.
Comment 5 Miguel de Icaza [MSFT] 2014-04-15 22:07:42 UTC
Ah, never mind.   I see that the comment say "either" grab the source, or build the binary.
Comment 6 Miguel de Icaza [MSFT] 2014-04-15 22:56:50 UTC
Steps to try this out from source:

$ git clone https://github.com/EventStore/EventStore.git

With Mono in your path, run:

$ run build.sh full

Open two terminals. 

On the first do:

$ mono bin/eventstore/release/anycpu/EventStore.SingleNode.exe 

On the other one do:

$ mono bin/eventstore/release/anycpu/EventStore.TestClient.exe 

Then when you get a prompt, type:

>>> wrfl 10 50000000

Apparently this takes a while.

I am running now with sgen, and will try with boehm.

While there is a message saying "Do not use Boehm", there are no reasons detailing why.
Comment 7 Mark Probst 2014-04-17 14:09:33 UTC
Greg,

Would you mind trying with mono master and setting the environment variable

  MONO_GC_DEBUG=clear-at-gc

I've been able to reproduce the bug somewhat reliably with default setting, but not with `clear-at-gc`.  If you can't either, I'll look into the detailed cause.
Comment 8 James Nugent 2014-04-18 10:55:12 UTC
Mark,

We've done some testing with `MONO_GC_DEBUG=clear-at-gc` on both OS X and Linux and can't reproduce the bug with it set.
Comment 9 James Nugent 2014-06-02 15:08:51 UTC
Hi,

Is there any progress on this?
Comment 10 Mark Probst 2014-08-05 15:36:27 UTC
James,

We have reason to believe that this issue is now fixed in mono master.  Would you mind trying to reproduce with master on default settings?
Comment 11 James Nugent 2014-08-05 16:45:06 UTC
Mark,

I'll switch one of our test clusters over to run mono from the master branch and report back.
Comment 12 James Nugent 2015-07-22 15:33:27 UTC
It appears this is still an issue even with master.

We've done a lot of stress testing recently and discovered there are still segmentation faults occurring even with clear-at-gc set, though less frequently - for example:

```
Stacktrace:


Native stacktrace:

	./clusternode() [0x61be82]
	./clusternode() [0x5c837b]
	./clusternode() [0x4656d3]
	/lib64/libpthread.so.0(+0xf130) [0x7f9b074fe130]
	./clusternode() [0x53693f]
	./clusternode() [0x53d107]
	./clusternode() [0x570d3e]
	/lib64/libpthread.so.0(+0x7df3) [0x7f9b074f6df3]
	/lib64/libc.so.6(clone+0x6d) [0x7f9b072241bd]

Debug info from gdb:

[New LWP 19760]
[New LWP 19737]
[New LWP 19696]
[New LWP 19620]
[New LWP 4949]
[New LWP 4040]
[New LWP 2730]
[New LWP 2655]
[New LWP 2236]
[New LWP 2217]
[New LWP 2216]
[New LWP 2215]
[New LWP 2214]
[New LWP 2213]
[New LWP 2211]
[New LWP 2210]
[New LWP 2209]
[New LWP 2207]
[New LWP 2206]
[New LWP 2205]
[New LWP 2203]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Missing separate debuginfo for /lib64/libgcc_s.so.1
Try: yum --enablerepo='*debug*' install /usr/lib/debug/.build-id/c5/2958e393bdf8e8d090f36de0f4e620d8736fbf.debug
0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  Id   Target Id         Frame 
  22   Thread 0x7f9aff3ff700 (LWP 2203) "Finalizer" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  21   Thread 0x7f9afefff700 (LWP 2205) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  20   Thread 0x7f9afedfe700 (LWP 2206) "Timer-Scheduler" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  19   Thread 0x7f9b00977700 (LWP 2207) "Threadpool moni" 0x00007f9b074fdce9 in waitpid () from /lib64/libpthread.so.0
  18   Thread 0x7f9afdfff700 (LWP 2209) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  17   Thread 0x7f9afddfe700 (LWP 2210) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  16   Thread 0x7f9afdbfd700 (LWP 2211) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  15   Thread 0x7f9afd7fb700 (LWP 2213) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  14   Thread 0x7f9afd5fa700 (LWP 2214) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  13   Thread 0x7f9afd3f9700 (LWP 2215) "clusternode" 0x00007f9b074fc7c0 in sem_wait () from /lib64/libpthread.so.0
  12   Thread 0x7f9aff17b700 (LWP 2216) "clusternode" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  11   Thread 0x7f9afe97b700 (LWP 2217) "IO Threadpool w" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  10   Thread 0x7f9a890ff700 (LWP 2236) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  9    Thread 0x7f9a8b773700 (LWP 2655) "IO Threadpool w" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  8    Thread 0x7f9a8b573700 (LWP 2730) "IO Threadpool w" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  7    Thread 0x7f9afc3d7700 (LWP 4040) "IO Threadpool w" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  6    Thread 0x7f9ae659f700 (LWP 4949) "IO Threadpool w" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  5    Thread 0x7f9a88cfd700 (LWP 19620) "Threadpool work" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  4    Thread 0x7f9a88efe700 (LWP 19696) "Threadpool work" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
  3    Thread 0x7f9afebfd700 (LWP 19737) "Threadpool work" 0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
  2    Thread 0x7f9a888fb700 (LWP 19760) "Threadpool work" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
* 1    Thread 0x7f9b085f8800 (LWP 2202) "clusternode" 0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6

Thread 22 (Thread 0x7f9aff3ff700 (LWP 2203)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9aff3fe7c0, info=0x7f9af80008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9aff3fe7c0) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fc7be in sem_wait () from /lib64/libpthread.so.0
#5  0x000000000056f357 in mono_sem_wait (sem=sem@entry=0x1d20f40 <finalizer_sem>, alertable=alertable@entry=1) at mono-semaphore.c:101
#6  0x00000000004a0e25 in finalizer_thread (unused=<optimized out>) at gc.c:1077
#7  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#8  start_wrapper (data=<optimized out>) at threads.c:710
#9  0x0000000000570d3e in inner_start_thread (arg=0x7ffd124d79d0) at mono-threads-posix.c:88
#10 0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f9afefff700 (LWP 2205)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afeffe400, info=0x7f9af00008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afeffe400) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b07215a0b in read () from /lib64/libc.so.6
#5  0x000000004027109a in ?? ()
#6  0x00007f9af0002650 in ?? ()
#7  0x00007f9afeffedc0 in ?? ()
#8  0x00007f9b00450430 in ?? ()
#9  0x0000000000000020 in ?? ()
#10 0x00007f9b0044f6d8 in ?? ()
#11 0x00007f9af00025f0 in ?? ()
#12 0x00000000401a12f8 in ?? ()
#13 0x00007f9afeffea60 in ?? ()
#14 0x00007f9afeffe9a0 in ?? ()
#15 0x0000000040270f40 in ?? ()
#16 0x00007f9b00450450 in ?? ()
#17 0x0000000040270ecf in ?? ()
#18 0x0000000000000800 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 20 (Thread 0x7f9afedfe700 (LWP 2206)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afedfd0c0, info=0x7f9af40008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afedfd0c0) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074faab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00000000005526a7 in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x100c, timeout=timeout@entry=0x7f9afedfd730, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1613
#6  0x0000000000564582 in WaitForSingleObjectEx (handle=0x100c, timeout=timeout@entry=49, alertable=alertable@entry=1) at wait.c:196
#7  0x000000000053ca2f in mono_wait_uninterrupted (thread=thread@entry=0x7f9b085558d0, multiple=multiple@entry=0, numhandles=numhandles@entry=1, handles=handles@entry=0x7f9afedfd7d8, waitall=waitall@entry=0, ms=ms@entry=49, alertable=1) at threads.c:1363
#8  0x000000000053dd8c in ves_icall_System_Threading_WaitHandle_WaitOne_internal (this=<optimized out>, handle=0x100c, ms=49, exitContext=<optimized out>) at threads.c:1496
#9  0x0000000040279d9e in ?? ()
#10 0x0000000000000011 in ?? ()
#11 0x0000000000000000 in ?? ()

Thread 19 (Thread 0x7f9b00977700 (LWP 2207)):
#0  0x00007f9b074fdce9 in waitpid () from /lib64/libpthread.so.0
#1  0x000000000061bf07 in mono_handle_native_sigsegv (signal=signal@entry=11, ctx=ctx@entry=0x7f9b08444ac0) at mini-exceptions.c:2323
#2  0x00000000005c837b in mono_arch_handle_altstack_exception (sigctx=sigctx@entry=0x7f9b08444ac0, fault_addr=<optimized out>, stack_ovf=stack_ovf@entry=0) at exceptions-amd64.c:861
#3  0x00000000004656d3 in mono_sigsegv_signal_handler (_dummy=11, _info=0x7f9b08444bf0, context=0x7f9b08444ac0) at mini.c:6858
#4  <signal handler called>
#5  0x000000000053693f in monitor_heuristic (tp=0x1d32500 <async_tp>, history=0x7f9ae8002650, history_size=<synthetic pointer>, current=<synthetic pointer>) at threadpool.c:846
#6  monitor_thread (unused=<optimized out>) at threadpool.c:955
#7  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#8  start_wrapper (data=<optimized out>) at threads.c:710
#9  0x0000000000570d3e in inner_start_thread (arg=0x7f9afedfd700) at mono-threads-posix.c:88
#10 0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#11 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f9afdfff700 (LWP 2209)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afdffdc80, info=0x7f9ac00008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afdffdc80) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fd87d in fsync () from /lib64/libpthread.so.0
#5  0x0000000000554611 in file_flush (handle=0x16c) at io.c:546
#6  0x0000000000623934 in ves_icall_System_IO_MonoIO_Flush (handle=<optimized out>, error=0x7f9afdffe2b8) at file-io.c:902
#7  0x000000004042562b in ?? ()
#8  0x00000009a1b8dd2f in ?? ()
#9  0x00000009a1b8df3f in ?? ()
#10 0x0000000000004c50 in ?? ()
#11 0x00007f9a12a65000 in ?? ()
#12 0x00007f9a12a65000 in ?? ()
#13 0x00007f9ac00025f0 in ?? ()
#14 0x0000000001b8dfbf in ?? ()
#15 0x000000614e3cc15f in ?? ()
#16 0x00007f9afdffe260 in ?? ()
#17 0x00000000404255a7 in ?? ()
#18 0x00007f99fc875a38 in ?? ()
#19 0x0000000000000000 in ?? ()

Thread 17 (Thread 0x7f9afddfe700 (LWP 2210)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afddfd300, info=0x7f9ab80008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afddfd300) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b07237ec4 in clock_nanosleep () from /lib64/libc.so.6
#5  0x0000000000565458 in SleepEx (ms=ms@entry=1, alertable=alertable@entry=1) at wthreads.c:277
#6  0x000000000053d99a in ves_icall_System_Threading_Thread_Sleep_internal (ms=1) at threads.c:1131
#7  0x00000000403317b3 in ?? ()
#8  0x00007f9af0002650 in ?? ()
#9  0x00007f9afddfddc0 in ?? ()
#10 0x0000000000000000 in ?? ()

Thread 16 (Thread 0x7f9afdbfd700 (LWP 2211)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afdbfc000, info=0x7f9abc0008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afdbfc000) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007ffd125d7d8c in gettimeofday ()
#5  0x000000000057248e in mono_100ns_datetime () at mono-time.c:173
#6  0x000000004019e6cd in ?? ()
#7  0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f9afd7fb700 (LWP 2213)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afd7fa000, info=0x7f9ab40008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afd7fa000) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074faab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00000000005526a7 in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x10f9, timeout=timeout@entry=0x7f9afd7fa660, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1613
#6  0x0000000000564582 in WaitForSingleObjectEx (handle=0x10f9, timeout=timeout@entry=100, alertable=alertable@entry=1) at wait.c:196
#7  0x000000000053ca2f in mono_wait_uninterrupted (thread=thread@entry=0x7f9aff17c170, multiple=multiple@entry=0, numhandles=numhandles@entry=1, handles=handles@entry=0x7f9afd7fa708, waitall=waitall@entry=0, ms=ms@entry=100, alertable=1) at threads.c:1363
#8  0x000000000053dd8c in ves_icall_System_Threading_WaitHandle_WaitOne_internal (this=<optimized out>, handle=0x10f9, ms=100, exitContext=<optimized out>) at threads.c:1496
#9  0x0000000040279d9e in ?? ()
#10 0x00007f9af0002650 in ?? ()
#11 0x00007f9afd7fadc0 in ?? ()
#12 0x00007f9b005a0eb8 in ?? ()
#13 0x00007f9aff1ca090 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x00007f9ab40025f0 in ?? ()
#16 0x0000000000000064 in ?? ()
#17 0x00007f9afd7fa7d0 in ?? ()
#18 0x00007f9afd7fa730 in ?? ()
#19 0x0000000040279adb in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 14 (Thread 0x7f9afd5fa700 (LWP 2214)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9afd5f9000, info=0x7f9aa80008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9afd5f9000) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074faab2 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x00000000005526a7 in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x10fa, timeout=timeout@entry=0x7f9afd5f9660, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1613
#6  0x0000000000564582 in WaitForSingleObjectEx (handle=0x10fa, timeout=timeout@entry=100, alertable=alertable@entry=1) at wait.c:196
#7  0x000000000053ca2f in mono_wait_uninterrupted (thread=thread@entry=0x7f9aff17c2d0, multiple=multiple@entry=0, numhandles=numhandles@entry=1, handles=handles@entry=0x7f9afd5f9708, waitall=waitall@entry=0, ms=ms@entry=100, alertable=1) at threads.c:1363
#8  0x000000000053dd8c in ves_icall_System_Threading_WaitHandle_WaitOne_internal (this=<optimized out>, handle=0x10fa, ms=100, exitContext=<optimized out>) at threads.c:1496
#9  0x0000000040279d9e in ?? ()
#10 0x00007f9af0002650 in ?? ()
#11 0x00007f9afd5f9dc0 in ?? ()
#12 0x00007f9b005a10b0 in ?? ()
#13 0x00007f9aff1eb3c8 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x00007f9aa80025f0 in ?? ()
#16 0x0000000000000064 in ?? ()
#17 0x00007f9afd5f97d0 in ?? ()
#18 0x00007f9afd5f9730 in ?? ()
#19 0x0000000040279adb in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 13 (Thread 0x7f9afd3f9700 (LWP 2215)):
#0  0x00007f9b074fc7c0 in sem_wait () from /lib64/libpthread.so.0
#1  0x000000000056f357 in mono_sem_wait (sem=0x1d2f760 <suspend_ack_semaphore>, alertable=alertable@entry=0) at mono-semaphore.c:101
#2  0x000000000052b70d in sgen_wait_for_suspend_ack (count=count@entry=14) at sgen-os-posix.c:188
#3  0x000000000052b831 in sgen_thread_handshake (suspend=suspend@entry=1) at sgen-os-posix.c:222
#4  0x0000000000530a27 in sgen_stop_world (generation=generation@entry=0) at sgen-stw.c:228
#5  0x0000000000522522 in sgen_perform_collection (requested_size=4096, generation_to_collect=0, reason=0x6e5655 "Nursery full", wait_to_finish=0) at sgen-gc.c:3173
#6  0x0000000000517303 in mono_gc_alloc_obj_nolock (vtable=vtable@entry=0x4dd8348, size=size@entry=32) at sgen-alloc.c:319
#7  0x00000000005173d5 in mono_gc_alloc_obj (vtable=vtable@entry=0x4dd8348, size=32) at sgen-alloc.c:500
#8  0x00000000004f0b21 in mono_object_allocate_spec (vtable=0x4dd8348, size=<optimized out>) at object.c:4397
#9  mono_object_new_alloc_specific (vtable=vtable@entry=0x4dd8348) at object.c:4497
#10 0x00000000004f0c10 in mono_object_new_specific (vtable=0x4dd8348) at object.c:4486
#11 0x00000000401a83c3 in ?? ()
#12 0x0000000000007433 in ?? ()
#13 0x0000000000007433 in ?? ()
#14 0x00007f9b00615a48 in ?? ()
#15 0x00007f9b00705d08 in ?? ()
#16 0x00007f9b00615a48 in ?? ()
#17 0x00007f9aac0025f0 in ?? ()
#18 0x0000000000007433 in ?? ()
#19 0x00007f9afd3f88f0 in ?? ()
#20 0x00007f9afd3f87a0 in ?? ()
#21 0x00000000402cd7e4 in ?? ()
#22 0x0000000000007433 in ?? ()
#23 0x00007f9b00615a48 in ?? ()
#24 0x00007f9b00705d08 in ?? ()
#25 0x00007f9b00615a48 in ?? ()
#26 0x00007f9b0091b378 in ?? ()
#27 0x00007f9b0068f8b8 in ?? ()
#28 0x0000000000007433 in ?? ()
#29 0x4fbe4254ebe46f5c in ?? ()
#30 0x402631a512020c9b in ?? ()
#31 0x0000000000007433 in ?? ()
#32 0x0000000000007433 in ?? ()
#33 0x00000000403444bc in ?? ()
#34 0x00007f9b0077e2c0 in ?? ()
#35 0x00007f9b0068f8b8 in ?? ()
#36 0x00007f9afd3f88f0 in ?? ()
#37 0x000000004045c1ff in ?? ()
#38 0x0000000000007433 in ?? ()
#39 0x00007f9b00615a48 in ?? ()
#40 0x00000009a1b86fdf in ?? ()
#41 0x00000009a1b86fdf in ?? ()
#42 0x00007f9b0068f8b8 in ?? ()
#43 0x00007f9b00705d08 in ?? ()
#44 0x0000000000000000 in ?? ()

Thread 12 (Thread 0x7f9aff17b700 (LWP 2216)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x00000000005370ea in tp_epoll_wait (p=0x1d325a0 <socket_io_data>) at ../../mono/metadata/tpool-epoll.c:121
#5  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#6  start_wrapper (data=<optimized out>) at threads.c:710
#7  0x0000000000570d3e in inner_start_thread (arg=0x7f9afdbfc290) at mono-threads-posix.c:88
#8  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f9afe97b700 (LWP 2217)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000536a4b in check_for_interruption_critical () at threadpool.c:1535
#5  0x000000000053912d in async_invoke_thread (data=0x0) at threadpool.c:1682
#6  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#7  start_wrapper (data=<optimized out>) at threads.c:710
#8  0x0000000000570d3e in inner_start_thread (arg=0x7f9afdbfc270) at mono-threads-posix.c:88
#9  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 10 (Thread 0x7f9a890ff700 (LWP 2236)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9a890fe300, info=0x7f9a840008e0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9a890fe300) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b07237ec4 in clock_nanosleep () from /lib64/libc.so.6
#5  0x0000000000565458 in SleepEx (ms=ms@entry=1, alertable=alertable@entry=1) at wthreads.c:277
#6  0x000000000053d99a in ves_icall_System_Threading_Thread_Sleep_internal (ms=1) at threads.c:1131
#7  0x00000000403317b3 in ?? ()
#8  0x00007f9af0002650 in ?? ()
#9  0x00007f9a890fedc0 in ?? ()
#10 0x00007f9aff1bcb50 in ?? ()
#11 0x00007f9aff1bcb50 in ?? ()
#12 0x0000000000000001 in ?? ()
#13 0x00007f9a840025f0 in ?? ()
#14 0x000000614e23d601 in ?? ()
#15 0x00007f9a890fea60 in ?? ()
#16 0x00007f9a890fe950 in ?? ()
#17 0x0000000040331611 in ?? ()
#18 0x00007f9aff1bcb50 in ?? ()
#19 0x00007f9aff1bcb50 in ?? ()
#20 0x00007f9a890fea60 in ?? ()
#21 0x00000000403ed683 in ?? ()
#22 0x00000000043444c8 in ?? ()
#23 0x00007f9a84001e80 in ?? ()
#24 0x00007f9af0002650 in ?? ()
#25 0x00007f9a890fe7d0 in ?? ()
#26 0x00007f9aff190e80 in ?? ()
#27 0x0000000001000068 in assembly_data_System_ServiceModel_dll ()
#28 0x00007f9aff1bcb50 in ?? ()
#29 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f9a8b773700 (LWP 2655)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9a8b7726c0, info=0x7f9aec110b80) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9a8b7726c0) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fcf7b in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#6  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=1) at sgen-gc.c:5328
#8  0x0000000000539101 in async_invoke_thread (data=0x0) at threadpool.c:1673
#9  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#10 start_wrapper (data=<optimized out>) at threads.c:710
#11 0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#12 0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#13 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 8 (Thread 0x7f9a8b573700 (LWP 2730)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000536a4b in check_for_interruption_critical () at threadpool.c:1535
#5  0x000000000053912d in async_invoke_thread (data=0x0) at threadpool.c:1682
#6  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#7  start_wrapper (data=<optimized out>) at threads.c:710
#8  0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#9  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 7 (Thread 0x7f9afc3d7700 (LWP 4040)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000536a4b in check_for_interruption_critical () at threadpool.c:1535
#5  0x000000000053912d in async_invoke_thread (data=0x0) at threadpool.c:1682
#6  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#7  start_wrapper (data=<optimized out>) at threads.c:710
#8  0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#9  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 6 (Thread 0x7f9ae659f700 (LWP 4949)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000536a4b in check_for_interruption_critical () at threadpool.c:1535
#5  0x000000000053912d in async_invoke_thread (data=0x0) at threadpool.c:1682
#6  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#7  start_wrapper (data=<optimized out>) at threads.c:710
#8  0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#9  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 5 (Thread 0x7f9a88cfd700 (LWP 19620)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000536a4b in check_for_interruption_critical () at threadpool.c:1535
#5  0x000000000053912d in async_invoke_thread (data=0x0) at threadpool.c:1682
#6  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#7  start_wrapper (data=<optimized out>) at threads.c:710
#8  0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#9  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#10 0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 4 (Thread 0x7f9a88efe700 (LWP 19696)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9a88efc780, info=0x7f9a0c002ed0) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9a88efc780) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00007f9b074ff4fc in _L_cond_lock_791 () from /lib64/libpthread.so.0
#6  0x00007f9b074ff3e7 in __pthread_mutex_cond_lock () from /lib64/libpthread.so.0
#7  0x00007f9b074fab84 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#8  0x00000000005526a7 in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x10f0, timeout=timeout@entry=0x7f9a88efce80, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1613
#9  0x0000000000564582 in WaitForSingleObjectEx (handle=0x10f0, timeout=timeout@entry=100, alertable=alertable@entry=1) at wait.c:196
#10 0x00000000004e7a85 in mono_monitor_try_enter_internal (allow_interruption=1, ms=4294967295, obj=0x7f9b08500fe0) at monitor.c:624
#11 ves_icall_System_Threading_Monitor_Monitor_try_enter_with_atomic_var (obj=0x7f9b08500fe0, ms=4294967295, lockTaken=0x7f9a88efcfd8 "") at monitor.c:1256
#12 0x00000000401a1511 in ?? ()
#13 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f9afebfd700 (LWP 19737)):
#0  0x00007f9b074fcf7d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#2  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000052349c in mono_gc_set_skip_thread (skip=skip@entry=0) at sgen-gc.c:5328
#4  0x0000000000539147 in async_invoke_thread (data=0x0) at threadpool.c:1686
#5  0x000000000053d107 in start_wrapper_internal (data=<optimized out>) at threads.c:663
#6  start_wrapper (data=<optimized out>) at threads.c:710
#7  0x0000000000570d3e in inner_start_thread (arg=0x7f9b00976c10) at mono-threads-posix.c:88
#8  0x00007f9b074f6df3 in start_thread () from /lib64/libpthread.so.0
#9  0x00007f9b072241bd in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f9a888fb700 (LWP 19760)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7f9a888f91c0, info=0x7f9ab011f440) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7f9a888f91c0) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fcf7b in __lll_lock_wait () from /lib64/libpthread.so.0
#5  0x00007f9b074f8d41 in _L_lock_790 () from /lib64/libpthread.so.0
#6  0x00007f9b074f8c47 in pthread_mutex_lock () from /lib64/libpthread.so.0
#7  0x00000000005173ca in mono_gc_alloc_obj (vtable=0x7f9ab0071478, size=56) at sgen-alloc.c:499
#8  0x000000004019cf93 in ?? ()
#9  0x00007f9b00400aa0 in ?? ()
#10 0x00007f9b00400a40 in ?? ()
#11 0x00007f9ae56481d0 in ?? ()
#12 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7f9b085f8800 (LWP 2202)):
#0  0x00007f9b07163962 in sigsuspend () from /lib64/libc.so.6
#1  0x000000000052b4be in suspend_thread (context=0x7ffd124d6f00, info=0x3d1f120) at sgen-os-posix.c:126
#2  suspend_handler (_dummy=<optimized out>, _info=<optimized out>, context=0x7ffd124d6f00) at sgen-os-posix.c:153
#3  <signal handler called>
#4  0x00007f9b074fa703 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#5  0x000000000055273b in _wapi_handle_timedwait_signal_handle (handle=handle@entry=0x100a, timeout=timeout@entry=0x0, alertable=alertable@entry=1, poll=poll@entry=0) at handles.c:1615
#6  0x00000000005527fb in _wapi_handle_wait_signal_handle (handle=handle@entry=0x100a, alertable=alertable@entry=1) at handles.c:1560
#7  0x000000000056474b in WaitForSingleObjectEx (handle=0x100a, timeout=timeout@entry=4294967295, alertable=alertable@entry=1) at wait.c:194
#8  0x000000000053ca2f in mono_wait_uninterrupted (thread=thread@entry=0x7f9b08554010, multiple=multiple@entry=0, numhandles=numhandles@entry=1, handles=handles@entry=0x7ffd124d75f8, waitall=waitall@entry=0, ms=ms@entry=-1, alertable=1) at threads.c:1363
#9  0x000000000053dd8c in ves_icall_System_Threading_WaitHandle_WaitOne_internal (this=<optimized out>, handle=0x100a, ms=-1, exitContext=<optimized out>) at threads.c:1496
#10 0x0000000040279d9e in ?? ()
#11 0x0000000003da33e0 in ?? ()
#12 0x00007ffd124d7bb0 in ?? ()
#13 0x00007f9b005a11d0 in ?? ()
#14 0x00007f9b084e2540 in ?? ()
#15 0xffffffffffffffff in ?? ()
#16 0x0000000003d89db0 in ?? ()
#17 0x00007f9b005a11d0 in ?? ()
#18 0x00007ffd124d76c0 in ?? ()
#19 0x00007ffd124d7620 in ?? ()
#20 0x0000000040279adb in ?? ()
#21 0x00007f9b084e2540 in ?? ()
#22 0x00007f9b005a11d0 in ?? ()
#23 0x0000000000000000 in ?? ()

=================================================================
Got a SIGSEGV while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries 
used by your application.
=================================================================

run-node.sh: Zeile 3:  2202 Abgebrochen             LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH:/opt/mono/lib 
```
Comment 13 Rodrigo Kumpera 2017-10-11 17:12:59 UTC
Can you still reproduce this bug with a recent mono, we fixed a few issues around the GC and the threadpool - which is what's crashing in your last report.

Note You need to log in before you can comment on or make changes to this bug.