Bug 19058

Summary: SIGSEGV signal in mono_signature_hash
Product: [Mono] Runtime Reporter: David Straw <david.straw>
Component: JITAssignee: Bugzilla <bugzilla>
Status: RESOLVED FIXED    
Severity: normal CC: matt.zinkevicius, mono-bugs+mono, mono-bugs+runtime, rusvdw, tspink, vargaz
Priority: ---    
Version: 3.2.x   
Target Milestone: ---   
Hardware: Other   
OS: Linux   
Tags: Is this bug a regression?: ---
Last known good build:

Description David Straw 2014-04-15 12:13:12 UTC
Description of Problem:
Our application is crashing under load due to a SIGSEGV with the following stack trace:

Thread 4 (Thread 0x7fd923bfd700 (LWP 16583)):
#0  0x00007fd96406b09d in waitpid () from /lib64/libpthread.so.0
#1  0x00000000004a54b4 in mono_handle_native_sigsegv ()
#2  0x000000000050105b in mono_arch_handle_altstack_exception ()
#3  0x0000000000415679 in mono_sigsegv_signal_handler ()
#4  <signal handler called>
#5  0x0000000000559bff in mono_signature_hash ()
#6  0x000000000062e7b9 in rehash ()
#7  0x000000000062ebd5 in monoeg_g_hash_table_insert_replace ()
#8  0x0000000000544e80 in mono_mb_create_and_cache ()
#9  0x000000000054b25c in mono_marshal_get_delegate_invoke ()
#10 0x00000000004aa05b in mono_delegate_trampoline ()
#11 0x0000000040fa7206 in ?? ()
#12 0x000000000000000b in ?? ()
#13 0x00000000004a780f in mono_create_delegate_trampoline_with_method ()
#14 0x0000000041180a37 in ?? ()
#15 0x0000000000000000 in ?? ()

If we build and install the Mono debuginfo and run in gdb our app crashes within seconds with various stack traces:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fb3ec981700 (LWP 10166)]
0x00000000005cb842 in mono_class_get_image (klass=0x52454645525f3836) at class.c:8378
8378            return klass->image;
(gdb) backtrace
#0  0x00000000005cb842 in mono_class_get_image (klass=0x52454645525f3836) at class.c:8378
#1  0x00000000005da0d6 in mono_debug_symfile_lookup_method (handle=0x7fb3c00f8eb0, method=Traceback (most recent call last):
  File "/usr/bin/mono-sgen-gdb.py", line 153, in to_string
    class_name = stringify_class_name (klass ["name_space"].string (), klass ["name"].string ())
RuntimeError: Cannot access memory at address 0x52454645525f387e
) at debug-mono-symfile.c:721
#2  0x00000000006414bd in lookup_method_func (key=0x7fb3c00f7d30, value=0x7fb3c00f8eb0, user_data=0x7fb3ec97f580) at mono-debug.c:490
#3  0x000000000074df8a in monoeg_g_hash_table_foreach (hash=0x1c11780, func=0x641469 <lookup_method_func>, user_data=0x7fb3ec97f580) at ghashtable.c:354
#4  0x0000000000641510 in _mono_debug_lookup_method (method=Traceback (most recent call last):
  File "/usr/bin/mono-sgen-gdb.py", line 153, in to_string
    class_name = stringify_class_name (klass ["name_space"].string (), klass ["name"].string ())
RuntimeError: Cannot access memory at address 0x52454645525f387e
) at mono-debug.c:504
#5  0x0000000000641533 in mono_debug_lookup_method (method=Traceback (most recent call last):
  File "/usr/bin/mono-sgen-gdb.py", line 153, in to_string
    class_name = stringify_class_name (klass ["name_space"].string (), klass ["name"].string ())
RuntimeError: Cannot access memory at address 0x52454645525f387e
) at mono-debug.c:521
#6  0x0000000000526582 in emit_all_line_number_info (w=0x7fb3b024c760) at dwarfwriter.c:722
#7  0x00000000005272f8 in mono_dwarf_writer_close (w=0x7fb3b024c760) at dwarfwriter.c:969
#8  0x00000000005461e5 in xdebug_end_emit (w=0x7fb3b024de70, dw=0x7fb3b024c760, method=0x0) at xdebug.c:210
#9  0x00000000005462c5 in mono_xdebug_flush () at xdebug.c:263
#10 0x0000000000546359 in mono_save_xdebug_info (cfg=0x7fb3c07fb7e0) at xdebug.c:293
#11 0x0000000000421421 in mini_method_compile (method="System.Linq.Enumerable:CreateSelectIterator ()", opts=370239999, domain=0x1c52aa0, flags=JIT_FLAG_RUN_CCTORS, parts=0) at mini.c:5713
#12 0x00000000004221db in mono_jit_compile_method_inner (method="System.Linq.Enumerable:CreateSelectIterator ()", target_domain=0x1c52aa0, opt=370239999, jit_ex=0x7fb3ec97fcb0) at mini.c:6013
#13 0x000000000042303d in mono_jit_compile_method_with_opt (method="System.Linq.Enumerable:CreateSelectIterator ()", opt=370239999, ex=0x7fb3ec97fcb0) at mini.c:6284
#14 0x000000000042321e in mono_jit_compile_method (method="System.Linq.Enumerable:CreateSelectIterator ()") at mini.c:6321
#15 0x000000000068cc66 in mono_compile_method (method="System.Linq.Enumerable:CreateSelectIterator ()") at object.c:587
#16 0x0000000000507fdb in common_call_trampoline (regs=0x7fb3ec9800e8, code=0x421511bc "L\213\064$L\213|$\bH\203\304\030\303\350\061\216\313\377\b \247v\300\263\177", m="System.Linq.Enumerable:CreateSelectIterator ()", tramp=
    0x421511ca "\350\061\216\313\377\b \247v\300\263\177", vt=0x0, vtable_slot=0x0, need_rgctx_tramp=0) at mini-trampolines.c:597
#17 0x0000000000508626 in mono_magic_trampoline (regs=0x7fb3ec9800e8, code=0x421511bc "L\213\064$L\213|$\bH\203\304\030\303\350\061\216\313\377\b \247v\300\263\177", arg=0x7fb3c076a720, tramp=
    0x421511ca "\350\061\216\313\377\b \247v\300\263\177") at mini-trampolines.c:717
#18 0x0000000041e0a166 in generic_trampoline_jit ()
#19 0x00000000421511bc in ?? ()
#20 0x00007fb3f0b76098 in ?? ()
#21 0x00007fb3bfb26100 in ?? ()
...

Another example stack trace:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fcbadafe700 (LWP 24881)]
0x0000000000528b48 in emit_line_number_info (w=0x7fcb88ad4e60, method=Traceback (most recent call last):
  File "/usr/bin/mono-sgen-gdb.py", line 153, in to_string
    class_name = stringify_class_name (klass ["name_space"].string (), klass ["name"].string ())
  File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcb in position 4: invalid continuation byte
, start_symbol=0x0, end_symbol=0x0, code=0x40e34560 "H\203\354\030L\211\064$H\211|$\bH\213D$\bH\213@\020\203x\030", code_size=174, debug_info=0x0) at dwarfwriter.c:1587
1587            ln_array = g_new0 (MonoDebugLineNumberEntry, debug_info->num_line_numbers);
(gdb) backtrace
#0  0x0000000000528b48 in emit_line_number_info (w=0x7fcb88ad4e60, method=Traceback (most recent call last):
  File "/usr/bin/mono-sgen-gdb.py", line 153, in to_string
    class_name = stringify_class_name (klass ["name_space"].string (), klass ["name"].string ())
  File "/usr/lib64/python2.6/encodings/utf_8.py", line 16, in decode
    return codecs.utf_8_decode(input, errors, True)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xcb in position 4: invalid continuation byte
, start_symbol=0x0, end_symbol=0x0, code=0x40e34560 "H\203\354\030L\211\064$H\211|$\bH\213D$\bH\213@\020\203x\030", code_size=174, debug_info=0x0) at dwarfwriter.c:1587
#1  0x0000000000526c10 in emit_all_line_number_info (w=0x7fcb88ad4e60) at dwarfwriter.c:827
#2  0x00000000005272f8 in mono_dwarf_writer_close (w=0x7fcb88ad4e60) at dwarfwriter.c:969
#3  0x00000000005461e5 in xdebug_end_emit (w=0x7fcb889c3700, dw=0x7fcb88ad4e60, method=0x0) at xdebug.c:210
#4  0x00000000005462c5 in mono_xdebug_flush () at xdebug.c:263
#5  0x0000000000546359 in mono_save_xdebug_info (cfg=0x7fcb88a7df20) at xdebug.c:293
#6  0x0000000000421421 in mini_method_compile (method="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute ()", opts=370239999, domain=0x22d0aa0, flags=JIT_FLAG_RUN_CCTORS, p                                                                   arts=
    0) at mini.c:5713
#7  0x00000000004221db in mono_jit_compile_method_inner (method="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute ()", target_domain=0x22d0aa0, opt=370239999, jit_ex=
    0x7fcbadafb7e0) at mini.c:6013
#8  0x000000000042303d in mono_jit_compile_method_with_opt (method="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute ()", opt=370239999, ex=0x7fcbadafb7e0) at mini.c:6284
#9  0x000000000042321e in mono_jit_compile_method (method="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute ()") at mini.c:6321
#10 0x000000000068cc66 in mono_compile_method (method="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute ()") at object.c:587
#11 0x0000000000507fdb in common_call_trampoline (regs=0x7fcbadafbc18, code=0x40e35244 "L\213\340I\213ą\300\017\204@", m="Mono.Xml.XPath.DTMXPathNavigator2:findAttribute                                                                    ()",
    tramp=0x40e352c7 "\350\064\255\237\377\b\240R\233\210\313\177", vt=0x0, vtable_slot=0x0, need_rgctx_tramp=0) at mini-trampolines.c:597
#12 0x0000000000508626 in mono_magic_trampoline (regs=0x7fcbadafbc18, code=0x40e35244 "L\213\340I\213ą\300\017\204@", arg=0x7fcb889b52a0, tramp=
    0x40e352c7 "\350\064\255\237\377\b\240R\233\210\313\177") at mini-trampolines.c:717
#13 0x0000000040830166 in generic_trampoline_jit ()
#14 0x0000000040e35244 in ?? ()
#15 0x00007fcbcf8f6100 in ?? ()
#16 0x00007fcbcfb204d0 in ?? ()
#17 0x00007fcbcf8d2f00 in ?? ()
#18 0x00007fcbae80ea10 in ?? ()
...

Inspecting the variables in the debug environment, we find that the pointer from a MethodLineNumberInfo to its 'method' field is pointing to corrupted or overwritten memory, which manifests as various errors such as memory access violations and UTF-8 encoding errors.

We are working on creating a manageable reproduction case. Right now the main issue (first stack trace above) only occurs in our full application running under load.
Comment 1 David Straw 2014-04-15 12:16:26 UTC
Forgot to mention that this is on CentOS 6.4 with Linux kernel 2.6.32-358.6.2.el6.x86_64. We have tried Mono 3.2.8, 3.4.0, and the latest in master as of Apr. 14, 2014 all with similar results.
Comment 2 Zoltan Varga 2014-04-15 23:17:08 UTC
These are probably two different problems, mono has a 'gdb mode' where it emits gdb debug info on the fly, this is rarely used so it might not be very stable. The first problem seems more serious.
Comment 3 Matt Z 2014-04-16 00:25:59 UTC
Looks like the identical SIGSEGV has been seen on the Mono build server when running unit tests as well:

http://wrench.mono-project.com/Wrench/WebServices/Download.aspx?workfile_id=2773144
http://wrench.mono-project.com/Wrench/WebServices/Download.aspx?workfile_id=2774714
Comment 4 David Straw 2014-04-16 15:18:51 UTC
@Zoltan Varga: Thanks for the info. If this path is rarely used, what is the typical way to debug the Mono runtime?

Agreed that the first problem is more serious. That's the issue that is crashing our application.
Comment 5 David Straw 2014-04-17 11:04:22 UTC
Below is a better stack trace with line numbers (Mono 3.4.0). I attached gdb and examined the variables at the time of the crash and found that the method signature that it was trying to hash reported having 35 parameters and 7 generic parameters.

The SIGSEGV occurred when attempting to dereference the second parameter because the pointer had a value of 0x25. The method in question is I believe a lambda that takes one parameter of a type defined in our code and returns a Guid.

The issue is most likely to occur whenever we emit IL during a web service call. We're not sure if the web service call really has anything to do with it but we couldn't reproduce the issue without it. We are using ServiceStack for our (REST) web service. The IL was being emitted by a call to AsQueryable, which apparently creates dynamic expressions that need to be JITted every time you call it. We plan to file a separate bug for that if it differs from .NET's behavior.

Stack trace:
(gdb) backtrace
#0  0x000000000061e749 in mono_type_hash (data=0x0) at metadata.c:1385
#1  0x0000000000624ba1 in mono_signature_hash (sig=0x7fc868187e10) at metadata.c:4852
#2  0x000000000074dc0e in do_rehash (hash=0x1dee140) at ghashtable.c:205
#3  0x000000000074dcf2 in rehash (hash=0x1dee140) at ghashtable.c:225
#4  0x000000000074dd73 in monoeg_g_hash_table_insert_replace (hash=0x1dee140, key=0x7fc868184670, value=0x7fc868180180, replace=0) at ghashtable.c:241
#5  0x0000000000601574 in mono_mb_create_and_cache (cache=0x1dee140, key=0x7fc868184670, mb=0x7fc8681833a0, sig=0x7fc84435dd30, max_stack=17) at marshal.c:2495
#6  0x0000000000606d75 in mono_marshal_get_delegate_invoke (method="System.Func`2:Invoke ()", del=0x7fc88174ff20) at marshal.c:4427
#7  0x0000000000509444 in mono_delegate_trampoline (regs=0x7fc87e591c48, code=0x0, arg=0x7fc8443688a0, tramp=0x41461ff8 "裀(\377\b\240\210\066D\310\177") at mini-trampolines.c:1177
#8  0x00000000406ea206 in ?? ()
#9  0x00007fc87e591a80 in ?? ()
#10 0x000000000073cf12 in mono_internal_hash_table_lookup (table=0xdcfab8, key=0xfffffffc) at mono-internal-hash.c:51
#11 0x00000000414626e7 in ?? ()
#12 0x00007fc87e591ef0 in ?? ()
#13 0x00007fc88174ff88 in ?? ()
#14 0x00007fc85d5bf080 in ?? ()
#15 0x00007fc88174ff20 in ?? ()
#16 0x00007fc88174ff88 in ?? ()
#17 0x00007fc88174ff88 in ?? ()
#18 0x0000000000000000 in ?? ()
Comment 6 Zoltan Varga 2014-04-20 04:17:05 UTC
Does the app which crashes uses appdomains/appdomain unloading ?
Comment 7 David Straw 2014-04-20 09:33:53 UTC
We have some dependencies that call AppDomain.CurrentDomain.DefineDynamicAssembly but I don't think we're loading/unloading any appdomains.
Comment 8 Tom Spink 2014-04-28 05:43:58 UTC
Hi,

I am pretty sure I am also having this problem (I've been having it for quite a while now), but my best efforts towards tracing it have come up fruitless as I can't reliably reproduce it.  It's only happening under heavy load.

My application is a multi-threaded web server (using HttpListener), where each request is processed in a separate thread.  Each thread has it's own PostgreSQL connection too, and database activity occurs in these threads in their own connections.

The problem happens when the number of users gets up to about 3/4, and there are many requests in quick succession.  Because it's non-deterministic, I'm unable to reliably reproduce the problem until the users start using the system - and, unfortunately, I'm not able to sit in front of the application running in GDB all day waiting for the issue to occur.  Incidentally, I disabled multi-threaded request handling, making all requests synchronous on the same thread, and the problem did not go away.

I'm not getting any useful information back from the native segfault (just code addresses), is this because I'm not running mono with the -debug flag?

However, using addr2line on two separate segfaults gives me a two different locations in mini-trampoline.c:

mini-trampolines.c:1177
mini-trampolines.c:82

Let me know what I can do to provide more useful information.

Thanks,
Tom.
Comment 9 Tom Spink 2014-05-08 05:38:57 UTC
OK, here's my latest stack trace:

#0  mono_type_hash (data=0x0) at metadata.c:1389
#1  0x000000000056c5a5 in mono_signature_hash (sig=0x7fffb4008a90) at metadata.c:4906
#2  0x0000000000636599 in do_rehash (hash=0x7fffc809eb50) at ghashtable.c:205
#3  rehash (hash=0x7fffc809eb50) at ghashtable.c:225
#4  0x0000000000636785 in monoeg_g_hash_table_insert_replace (hash=0x7fffc809eb50, key=0x7fffc81d4430, value=0x7fffb4010f68, replace=0) at ghashtable.c:241
#5  0x0000000000550d60 in mono_mb_create_and_cache_full (cache=cache@entry=0x7fffc809eb50, key=key@entry=0x7fffc81d4430, mb=mb@entry=0x7fffc81d9c80, sig=sig@entry=0x7fffc805cfc0, max_stack=17, out_found=out_found@entry=0x0) at marshal.c:2498
#6  0x0000000000557356 in mono_mb_create_and_cache (max_stack=<optimized out>, sig=0x7fffc805cfc0, mb=0x7fffc81d9c80, key=0x7fffc81d4430, cache=0x7fffc809eb50) at marshal.c:2517
#7  mono_marshal_get_delegate_invoke_internal (target_method=<optimized out>, static_method_with_first_arg_bound=1, callvirt=<optimized out>, method=<optimized out>) at marshal.c:4445
#8  mono_marshal_get_delegate_invoke (method=method@entry=0x7fffc805cf50, del=del@entry=0x7ffff6b24128) at marshal.c:4478
#9  0x00000000004ba1cc in mono_delegate_trampoline (regs=<optimized out>, code=<optimized out>, arg=<optimized out>, tramp=<optimized out>) at mini-trampolines.c:1177
#10 0x0000000040003206 in ?? ()
#11 0x0000000000000000 in ?? ()

In #1, data is NULL, which is obviously the cause of the segfault, so I did this:

(gdb) up
#1  0x000000000056c5a5 in mono_signature_hash (sig=0x7fffb4008a90) at metadata.c:4906
4906                    res = (res << 5) - res + mono_type_hash (sig->params[i]);
(gdb) p sig
$1 = (MonoMethodSignature *) 0x7fffb4008a90
(gdb) p *sig
$2 = {ret = 0x7ffff4409020, param_count = 49, sentinelpos = 0, generic_param_count = 494, call_convention = 0, hasthis = 0, explicit_this = 0, pinvoke = 0, is_inflated = 0, has_type_parameters = 0, params = 0x7fffb4008aa0}
(gdb) 

It looks like sig may be corrupt, as the param_count seems suspiciously large.  Is there anything else I can do to help narrow this problem down further?

Thanks,
Tom.
Comment 10 Zoltan Varga 2014-05-08 09:38:51 UTC
Without a reproducible testcase, it would be very hard to track this down.
Comment 11 Tom Spink 2014-05-09 11:27:46 UTC
Indeed, it's hard enough to trigger the issue manually - it seems to happen after a couple of hours, and when multiple people are using the system, therefore there are multiple threads running, with multiple database connections, and multiple nested LINQ expressions on the go.

I don't know what aspect of this may be the issue... if it's the LINQ expressions (and the JIT compilation of the various subexpressions/lambdas) then perhaps I can whip up a test program.  I'll try my best to get something reproducible.

Thanks,
Tom.
Comment 12 Russell van der Walt 2014-06-18 05:05:24 UTC
I'm also experiencing the same crash on a fastcgi web app. All was good on mono 2.6, but as soon as we upgraded to 3.4 (master/934b127) the app dies within minutes with a SIGSEGV with the following native stacktrace:

Native stacktrace:

	/usr/local/bin/mono() [0x4ac854]
	/usr/local/bin/mono() [0x50491f]
	/usr/local/bin/mono() [0x421177]
	/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0) [0x7f98a057acb0]
	/usr/local/bin/mono() [0x55fea0]
	/usr/local/bin/mono(mono_signature_hash+0x45) [0x5614e5]
	/usr/local/bin/mono() [0x62ff79]
	/usr/local/bin/mono() [0x630175]
	/usr/local/bin/mono() [0x544a30]
	/usr/local/bin/mono() [0x54b5ec]
	/usr/local/bin/mono() [0x4afce3]
	[0x41334206]

Luckily there is often a managed stacktrace too:

Stacktrace:

  at <unknown> <0xffffffff>
  at System.Linq.SortSequenceContext`2<Core.API.Model.CounterEntry, System.DateTime>.Initialize (Core.API.Model.CounterEntry[]) <0x000c4>
  at System.Linq.QuickSort`1<Core.API.Model.CounterEntry>.PerformSort () <0x0002c>
  at System.Linq.QuickSort`1/<Sort>c__Iterator0<Core.API.Model.CounterEntry>.MoveNext () <0x000de>
  at System.Linq.Enumerable.ToArray<Core.API.Model.CounterEntry> (System.Collections.Generic.IEnumerable`1<Core.API.Model.CounterEntry>) <0x00240>
  at System.Linq.Enumerable/<CreateReverseIterator>c__IteratorF`1<Core.API.Model.CounterEntry>.MoveNext () <0x0005b>
  at System.Linq.Enumerable.First<Core.API.Model.CounterEntry> (System.Collections.Generic.IEnumerable`1<Core.API.Model.CounterEntry>,System.Func`2<Core.API.Model.CounterEntry, bool>,System.Linq.Enumerable/Fallback) <0x00098>
  at System.Linq.Enumerable.FirstOrDefault<Core.API.Model.CounterEntry> (System.Collections.Generic.IEnumerable`1<Core.API.Model.CounterEntry>) <0x0003b>
  at (wrapper dynamic-method) System.Runtime.CompilerServices.ExecutionScope.lambda_method (System.Runtime.CompilerServices.ExecutionScope) <0x0013b>
  at System.Linq.QueryableEnumerable`1<Core.API.Model.CounterEntry>.Execute<Core.API.Model.CounterEntry> (System.Linq.Expressions.Expression) <0x0007f>
  at System.Linq.Queryable.FirstOrDefault<Core.API.Model.CounterEntry> (System.Linq.IQueryable`1<Core.API.Model.CounterEntry>) <0x00119>

Full log here: https://gist.github.com/rusvdw/f4d0ef9cf8d6632ca33f#file-log1

This has allowed me to isolate it to the Queryable.OrderBy method. The following code when run under load in a web app (using fastcgi/nginx) will cause the same SIGSEGV:

    class TestClass {
        public DateTime date;
    }

    protected void Page_Load(object sender, EventArgs e)
    {

        var queryable = new List<TestClass>() {
                new TestClass() { date = new DateTime(1930, 1, 1) },
                new TestClass() { date = new DateTime(1910, 1, 1) },
                new TestClass() { date = new DateTime(1920, 1, 1) }
            }.AsQueryable();


        queryable.OrderBy(x => x.date).ToList(); // causes SIGSEGV

        Response.ContentType = "text/plain";
        Response.Write("ok");
        Response.End();

    }

fastcgi-mono-server2 was run as follows:

fastcgi-mono-server2 /applications=/:/var/www/test/ /socket=tcp:127.0.0.1:9000 --verbose &>/var/log/core/fastcgi--$(date +%Y%m%d-%H%M%S).log &

Add some load to the app using a load testing tool and it should die within a couple of seconds. Full error log here: https://gist.github.com/rusvdw/f4d0ef9cf8d6632ca33f#file-log2

A workaround seems to be to convert the Queryable to a List first:

  queryable.ToList().OrderBy(x => x.date).ToList(); // has no issues

I was not able to reproduce this in a console application.

Environment details:

Ubuntu 12.04.4 LTS (GNU/Linux 3.14.5-x86_64-linode42 x86_64)

Mono JIT compiler version 3.4.1 (master/934b127 Fri Jun 13 08:34:19 UTC 2014)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
        TLS:           __thread
        SIGSEGV:       altstack
        Notifications: epoll
        Architecture:  amd64
        Disabled:      none
        Misc:          softdebug
        LLVM:          supported, not enabled.
        GC:            sgen
Comment 13 Matt Z 2014-06-18 15:14:10 UTC
Like Russell, our team has also mostly worked-around this issue by materializing our QueryableEnumerable instances by using .ToList(). My hunch, given our stack traces, is that the JIT is puking on the dynamic code created by Queryable's.
Comment 14 Zoltan Varga 2014-06-19 11:07:09 UTC
Testcase:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
using System;
using System.Threading;
using System.Collections.Generic;
using System.Linq;

public class Tests
{
	class TestClass {
        public DateTime date;
    }

	public static void Main (String[] args) {
		for (int i = 0; i < 1000; ++i) {
			AppDomain ad = AppDomain.CreateDomain ("ad" + i);
			ad.DoCallBack (new CrossAppDomainDelegate (CB));
			AppDomain.Unload (ad);
			Console.WriteLine ("1");
		}
	}

	static void CB () {
		for (int i = 0; i < 100; ++i) {
			var queryable = new List<TestClass>() {
                new TestClass() { date = new DateTime(1930, 1, 1) },
                new TestClass() { date = new DateTime(1910, 1, 1) },
                new TestClass() { date = new DateTime(1920, 1, 1) }
            }.AsQueryable();

			queryable.OrderBy(x => x.date).ToList(); // causes SIGSEG
		}
	}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Comment 15 Russell van der Walt 2014-06-19 12:43:37 UTC
Using Zoltan's test case I have determined that the first branch that contains the SIGSEG is 3.2.3. Hopefully that is useful to someone.
Comment 16 Zoltan Varga 2014-06-19 14:59:41 UTC
Should be fixed by mono master 15c725c674d915c9463e891957e5149dcb23c844:

https://github.com/mono/mono/commit/15c725c674d915c9463e891957e5149dcb23c844

Please reopen if this still happens with that revision.
Thanks for the original testcase.
Comment 17 Russell van der Walt 2014-06-20 07:01:18 UTC
Thank you Zoltan, I can confirm that it has solved the issue.