Bug 17909 - System.Web tests crash randomly
Summary: System.Web tests crash randomly
Status: RESOLVED NOT_REPRODUCIBLE
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: unspecified
Hardware: PC Mac OS
: High major
Target Milestone: ---
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2014-02-20 10:41 UTC by Zoltan Varga
Modified: 2018-02-23 21:22 UTC (History)
4 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED NOT_REPRODUCIBLE

Description Zoltan Varga 2014-02-20 10:41:38 UTC
http://wrench.mono-project.com/Wrench/WebServices/Download.aspx?workfile_id=3028117

The crash happens here:

#6  mono_metadata_str_hash [inlined] () at /private/tmp/source/bockbuild/profiles/mono-mac-release/build-root/mono-3.2.7/mono/metadata/metadata.c:4542
#7  0x00245aa8 in mono_metadata_type_hash (t1=0x7c515854) at metadata.c:4582
#8  0x002a6287 in do_rehash (_data=0xb068d940) at mono-hash.c:216
#9  0x002d2c85 in mono_gc_invoke_with_gc_lock (func=0x2a6240 <do_rehash>, data=0xb068d940) at sgen-gc.c:4756
#10 0x002a694c in rehash (hash=0x21efb1) at mono-hash.c:243
#11 0x002a69c7 in mono_g_hash_table_insert_replace (hash=0xb068d998, key=0x7d2dd1f4, value=0x2f31b40, replace=0) at mono-hash.c:435
#12 0x002a6adf in mono_g_hash_table_insert (h=0x7be4e430, k=0x7d2dd1f4, v=0x2f31b40) at mono-hash.c:462
#13 0x002c2590 in mono_type_get_object (domain=0x7be4d930, type=0x7d2dd1f4) at reflection.c:6529

Here 't1' is the type of MonoTests.SystemWeb.Framework.WebTest/UnloadHandler, and it looks like this type/class is added to domain->type_hash for two difference domains, so when one is unloaded, there is stale data in the type_hash of the other.
Comment 1 Zoltan Varga 2014-03-20 16:49:23 UTC
The type is added first to type_hash for the child domain from:

<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
"TestRunnerThread" tid=0x0xb061f000 this=0x0x5f33450 thread handle 0x117 state : not waiting owns ()
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.Assembly.InternalGetType (System.Reflection.Assembly,System.Reflection.Module,string,bool,bool) <IL 0x00032, 0xffffffff>
  at System.Reflection.Assembly.GetType (string,bool,bool) [0x00028] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MonoAssembly.cs:61
  at System.Reflection.Assembly.GetType (string) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/Assembly.cs:365
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.GetDeserializationType (long,string,bool) [0x00065] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:869
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadType (System.IO.BinaryReader,System.Runtime.Serialization.Formatters.Binary.TypeTag,bool) [0x000cb] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:917
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadTypeMetadata (System.IO.BinaryReader,bool,bool) [0x0007b] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:631
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectInstance (System.IO.BinaryReader,bool,bool,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x0000a] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:265
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x00088] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:187
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObject (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,long&,object&,System.Runtime.Serialization.SerializationInfo&) [0x00103] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:219
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadNextObject (System.IO.BinaryReader) [0x00027] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:150
  at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ReadObjectGraph (System.Runtime.Serialization.Formatters.Binary.BinaryElement,System.IO.BinaryReader,bool,object&,System.Runtime.Remoting.Messaging.Header[]&) [0x0004d] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/ObjectReader.cs:107
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.NoCheckDeserialize (System.IO.Stream,System.Runtime.Remoting.Messaging.HeaderHandler) [0x0007a] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:177
  at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize (System.IO.Stream) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Serialization.Formatters.Binary/BinaryFormatter.cs:134
  at System.Runtime.Remoting.RemotingServices.DeserializeCallData (byte[]) [0x0000f] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting/RemotingServices.cs:794
  at (wrapper xdomain-dispatch) System.AppDomain.add_DomainUnload (object,byte[]&,byte[]&) <IL 0x00063, 0xffffffff>
  at (wrapper xdomain-invoke) System.AppDomain.add_DomainUnload (System.EventHandler) <0xffffffff>
  at (wrapper remoting-invoke-with-check) System.AppDomain.add_DomainUnload (System.EventHandler) <0xffffffff>
  at MonoTests.SystemWeb.Framework.WebTest.SetupHosting (MonoTests.SystemWeb.Framework.WebTestResourcesSetupAttribute/SetupHandler) <0x003ab>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

The second time, it is added to the main domain from:
##################################################################
"Threadpool worker" tid=0x0xb0bab000 this=0x0x6f3b030 thread handle 0x16c state : not waiting owns ()
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MethodBase.GetMethodFromHandleInternalType (intptr,intptr) <IL 0x00022, 0xffffffff>
  at System.Reflection.MethodBase.GetMethodFromIntPtr (intptr,intptr) [0x0001b] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MethodBase.cs:62
  at System.Reflection.MethodBase.GetMethodFromHandle (System.RuntimeMethodHandle) [0x00000] in /Users/vargaz/git/mono/mcs/class/corlib/System.Reflection/MethodBase.cs:70
  at System.Runtime.Remoting.Messaging.CADMethodCallMessage.GetMethod () [0x00036] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Messaging/CADMessages.cs:367
  at System.Runtime.Remoting.Messaging.MethodCall..ctor (System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x0004e] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Messaging/MethodCall.cs:90
  at System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) [0x00018] in /Users/vargaz/git/mono/mcs/class/corlib/System/AppDomain.cs:1371
  at (wrapper remoting-invoke-with-check) System.AppDomain.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage,byte[]&,System.Runtime.Remoting.Messaging.CADMethodReturnMessage&) <IL 0x0003d, 0xffffffff>
  at System.Runtime.Remoting.Channels.CrossAppDomainSink.ProcessMessageInDomain (byte[],System.Runtime.Remoting.Messaging.CADMethodCallMessage) [0x00008] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Channels/CrossAppDomainChannel.cs:199
  at (wrapper runtime-invoke) <Module>.runtime_invoke_CrossAppDomainSink/ProcessMessageRes_object_object (object,intptr,intptr,intptr) <IL 0x00064, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke (System.Reflection.MonoMethod,object,object[],System.Exception&) <IL 0x00030, 0xffffffff>
  at System.AppDomain.InvokeInDomainByID (int,System.Reflection.MethodInfo,object,object[]) [0x00017] in /Users/vargaz/git/mono/mcs/class/corlib/System/AppDomain.cs:971
  at System.Runtime.Remoting.Channels.CrossAppDomainSink.SyncProcessMessage (System.Runtime.Remoting.Messaging.IMessage) [0x0002e] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Channels/CrossAppDomainChannel.cs:235
  at System.Runtime.Remoting.Proxies.RemotingProxy.Invoke (System.Runtime.Remoting.Messaging.IMessage) [0x000e0] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Proxies/RemotingProxy.cs:100
  at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke (System.Runtime.Remoting.Proxies.RealProxy,System.Runtime.Remoting.Messaging.IMessage,System.Exception&,object[]&) [0x0009c] in /Users/vargaz/git/mono/mcs/class/corlib/System.Runtime.Remoting.Proxies/RealProxy.cs:194
  at (wrapper runtime-invoke) <Module>.runtime_invoke_object_object_object_intptr&_intptr& (object,intptr,intptr,intptr) <IL 0x0006e, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_remoting_wrapper (intptr,intptr) <0xffffffff>
  at (wrapper remoting-invoke) MonoTests.SystemWeb.Framework.WebTest/UnloadHandler.OnUnload (object,System.EventArgs) <0xffffffff>
  at System.AppDomain.DoDomainUnload () <0x00038>
  at (wrapper runtime-invoke) object.runtime_invoke_void__this__ (object,intptr,intptr,intptr) <IL 0x0004e, 0xffffffff>
  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) System.AppDomain.InternalUnload (int) <0xffffffff>
  at System.AppDomain.Unload (System.AppDomain) <0x0002b>
  at System.Web.HttpRuntime.DoUnload () <0x00017>
  at System.Web.HttpRuntime.<ShutdownAppDomain>m__2 (object) <0x0000b>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <IL 0x00052, 0xffffffff>
#############################################################################

So what happens is that AppDomain.DoDomainUnload () tries to invoke the Unload callback, which is from the main domain, so it does a remoting call, and the remoting code, running in the main domain, calls 
GetMethodFromHandle with a handle which refers to the UnloadHandler method from an assembly which is only loaded in the child domain. This causes a reference to a type from this assembly to be added to the main domain's type_hash, leading to a crash later when the child domain is unloaded.
Comment 2 Zoltan Varga 2014-03-20 21:00:14 UTC
Testcase:
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
using System;
using System.Reflection;

public class Example
{
  public static void Main(String[] args)
  {
	  for (int i = 0; i < 100; ++i) {
	  var setup = new AppDomainSetup ();
	  setup.ApplicationBase = "foo";
      var domain = AppDomain.CreateDomain("ChildDomain ", null, setup);
	  domain.Load (new AssemblyName ("bugu"));
	  UnloadHandler h = new UnloadHandler ();
	  domain.DomainUnload += h.OnUnload;
	  AppDomain.Unload (domain);

	  foreach (var t in typeof (int).Assembly.GetTypes ())
		  ;
	  }
  }
}

public class UnloadHandler :  MarshalByRefObject {
	public void OnUnload (object sender, EventArgs e) {
		Console.WriteLine ("HIT!");
	}
}
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Compile this into bug.exe, create a directory named 'foo', and copy bugu.exe into it, enable DEBUG_DOMAIN_UNLOAD in domain.c, then run bugu.exe from another dir. It should crash randomly after a few iterations. The type information is stored in image mempools, and it takes some time before those are overwritten in a way that makes mono_metadata_type_equal () to crash.
Comment 3 Ludovic Henry 2018-02-23 21:22:23 UTC
I can't reproduce with latest version of Mono 5.13.0.220 (master/798c5efa52a).