Bug 17909 - System.Web tests crash randomly
Summary: System.Web tests crash randomly
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: unspecified
Hardware: PC Mac OS
: High major
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2014-02-20 10:41 UTC by Zoltan Varga
Modified: 2014-04-10 14:29 UTC (History)
3 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Zoltan Varga 2014-02-20 10:41:38 UTC
http://wrench.mono-project.com/Wrench/WebServices/Download.aspx?workfile_id=3028117

The crash happens here:

#6  mono_metadata_str_hash [inlined] () at /private/tmp/source/bockbuild/profiles/mono-mac-release/build-root/mono-3.2.7/mono/metadata/metadata.c:4542
#7  0x00245aa8 in mono_metadata_type_hash (t1=0x7c515854) at metadata.c:4582
#8  0x002a6287 in do_rehash (_data=0xb068d940) at mono-hash.c:216
#9  0x002d2c85 in mono_gc_invoke_with_gc_lock (func=0x2a6240 <do_rehash>, data=0xb068d940) at sgen-gc.c:4756
#10 0x002a694c in rehash (hash=0x21efb1) at mono-hash.c:243
#11 0x002a69c7 in mono_g_hash_table_insert_replace (hash=0xb068d998, key=0x7d2dd1f4, value=0x2f31b40, replace=0) at mono-hash.c:435
#12 0x002a6adf in mono_g_hash_table_insert (h=0x7be4e430, k=0x7d2dd1f4, v=0x2f31b40) at mono-hash.c:462
#13 0x002c2590 in mono_type_get_object (domain=0x7be4d930, type=0x7d2dd1f4) at reflection.c:6529

Here 't1' is the type of MonoTests.SystemWeb.Framework.WebTest/UnloadHandler, and it looks like this type/class is added to domain->type_hash for two difference domains, so when one is unloaded, there is stale data in the type_hash of the other.
Comment 1 Zoltan Varga 2014-03-20 16:49:23 UTC
The type is added first to type_hash for the child domain from:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
"TestRunnerThread" tid=0x0xb061f000 this=0x0x5f33450 thread handle 0x117 state : not waiting owns ()
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.Assembly.InternalGetType (System.Reflection.Assembly,System.Reflection.Module,string,bool,bool) <IL 0x00032, 0xffffffff>
  at System.Reflection.Assembly.GetType (string,bool,bool) [0x00028] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MonoAssembly.cs:61
  at System.Reflection.Assembly.GetType (string) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/Assembly.cs:365
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetDeserializationType (long,string,bool) [0x00065] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:869
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadType (System.IO.BinaryReader,System.Runtime.Serialization.Formatters.Binary.TypeTag,bool) [0x000cb] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:917
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadTypeMetadata (System.IO.BinaryReader,bool,bool) [0x0007b] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:631
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectInstance (System.IO.BinaryReader,bool,bool,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x0000a] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:265
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x00088] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:187
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x00103] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:219
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadNextObject (System.IO.BinaryReader) [0x00027] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:150
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectGraph (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,bool,object&,System.Runtime.Remoting.Messaging.Header[]&) [0x0004d] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:107
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.NoCheckDeserialize (System.IO.Stream,System.Runtime.Remoting.Messaging.HeaderHandler) [0x0007a] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:177
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:134
  at System.Runtime.Remoting.RemotingServices.DeserializeCallData (byte[]) [0x0000f] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting/RemotingServices.cs:794
  at (wrapper xdomain-dispatch) System.AppDomain.add_DomainUnload (object,byte[]&,byte[]&) <IL 0x00063, 0xffffffff>
  at (wrapper xdomain-invoke) System.AppDomain.add_DomainUnload (System.EventHandler) <0xffffffff>
  at (wrapper remoting-invoke-with-check) System.AppDomain.add_DomainUnload (System.EventHandler) <0xffffffff>
  at MonoTests.SystemWeb.Framework.WebTest.SetupHosting (MonoTests.SystemWeb.Framework.WebTestResourcesSetupAttribute/SetupHandler) <0x003ab>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

The second time, it is added to the main domain from:
##################################################################
"Threadpool worker" tid=0x0xb0bab000 this=0x0x6f3b030 thread handle 0x16c state : not waiting owns ()
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MethodBase.GetMethodFromHandleInternalType (intptr,intptr) <IL 0x00022, 0xffffffff>
  at System.Reflection.MethodBase.GetMethodFromIntPtr (intptr,intptr) [0x0001b] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MethodBase.cs:62
  at System.Reflection.MethodBase.GetMethodFromHandle (System.RuntimeMethodHandle) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MethodBase.cs:70
  at System.Runtime.Remoting.Messaging.CADMethodCallMessage.GetMethod () [0x00036] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Messaging/CADMessages.cs:367
  at System.Runtime.Remoting.Messaging.MethodCall..ctor (System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x0004e] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Messaging/MethodCall.cs:90
  at System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) [0x00018] in /Users/vargaz/git/mono/mcs/class/corlib/System/AppDomain.cs:1371
  at (wrapper remoting-invoke-with-check) System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) <IL 0x0003d, 0xffffffff>
  at System.Runtime.Remoting.Channels.CrossAppDomainSink.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x00008] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Channels/CrossAppDomainChannel.cs:199
  at (wrapper runtime-invoke) <Module>.runtime_invoke_CrossAppDomainSink/ProcessMessageRes_object_object (object,intptr,intptr,intptr) <IL 0x00064, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) <IL 0x00030, 0xffffffff>
  at System.AppDomain.InvokeInDomainByID (int,System.Reflection.MethodInfo,object,object[]) [0x00017] in /Users/vargaz/git/mono/mcs/class/corlib/System/AppDomain.cs:971
  at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage (System.Runtime.Remoting.Messaging.IMessage) [0x0002e] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Channels/CrossAppDomainChannel.cs:235
  at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke (System.Runtime.Remoting.Messaging.IMessage) [0x000e0] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Proxies/RemotingProxy.cs:100
  at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke (System.Runtime.Remoting.Proxies.RealProxy,System.Runtime.Remoting.Messaging.IMessage,System.Exception&,object[]&) [0x0009c] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Proxies/RealProxy.cs:194
  at (wrapper runtime-invoke) <Module>.runtime_invoke_object_object_object_intptr&_intptr& (object,intptr,intptr,intptr) <IL 0x0006e, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_remoting_wrapper (intptr,intptr) <0xffffffff>
  at (wrapper remoting-invoke) MonoTests.SystemWeb.Framework.WebTest/UnloadHandler.OnUnload (object,System.EventArgs) <0xffffffff>
  at System.AppDomain.DoDomainUnload () <0x00038>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <IL 0x0004e, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.AppDomain.InternalUnload (int) <0xffffffff>
  at System.AppDomain.Unload (System.AppDomain) <0x0002b>
  at System.Web.HttpRuntime.DoUnload () <0x00017>
  at System.Web.HttpRuntime.<ShutdownAppDomain>m__2 (object) <0x0000b>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
#############################################################################

So what happens is that AppDomain.DoDomainUnload () tries to invoke the Unload callback, which is from the main domain, so it does a remoting call, and the remoting code, running in the main domain, calls 
GetMethodFromHandle with a handle which refers to the UnloadHandler method from an assembly which is only loaded in the child domain. This causes a reference to a type from this assembly to be added to the main domain's type_hash, leading to a crash later when the child domain is unloaded.
Comment 2 Zoltan Varga 2014-03-20 21:00:14 UTC
Testcase:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
using System;
using System.Reflection;

public class Example
{
  public static void Main(String[] args)
  {
	  for (int i = 0; i < 100; ++i) {
	  var setup = new AppDomainSetup ();
	  setup.ApplicationBase = "foo";
      var domain = AppDomain.CreateDomain("ChildDomain ", null, setup);
	  domain.Load (new AssemblyName ("bugu"));
	  UnloadHandler h = new UnloadHandler ();
	  domain.DomainUnload += h.OnUnload;
	  AppDomain.Unload (domain);

	  foreach (var t in typeof (int).Assembly.GetTypes ())
		  ;
	  }
  }
}

public class UnloadHandler :  MarshalByRefObject {
	public void OnUnload (object sender, EventArgs e) {
		Console.WriteLine ("HIT!");
	}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Compile this into bug.exe, create a directory named 'foo', and copy bugu.exe into it, enable DEBUG_DOMAIN_UNLOAD in domain.c, then run bugu.exe from another dir. It should crash randomly after a few iterations. The type information is stored in image mempools, and it takes some time before those are overwritten in a way that makes mono_metadata_type_equal () to crash.

Note You need to log in before you can comment on or make changes to this bug.