Bug 4587

Summary: aot deadlock while debugging
Product: iOS Reporter: Chris Toshok <toshok>
Component: XI runtimeAssignee: Zoltan Varga <vargaz>
Severity: normal CC: matt, miguel, mono-bugs+monotouch, rolf, tbolinger
Priority: High    
Version: 5.2   
Target Milestone: Untriaged   
Hardware: PC   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: test case
stacktrace showing deadlock
another deadlock stacktrace

Description Chris Toshok 2012-04-22 20:53:03 UTC

1) start debugging app on device
2) wait some random amount of time (sometimes it's pretty quick, other times I've had to wait a few minutes)
3) app becomes completely unresponsive.

I don't have to be doing anything with the app.  I have breakpoints registered but none of them are hit (nor should they be, they're in parts of the app I'm not touching.)

i see this in xcode's console log:

Apr 22 17:40:48 unknown com.apple.debugserver-64[7463] <Warning>: debugserver-64 for armv6 Copyright (c) 2007-2009 Apple, Inc.  All Rights Reserved.
Apr 22 17:40:48 unknown com.apple.debugserver-64[7463] <Warning>: Connecting to com.apple.debugserver service...
Apr 22 17:40:48 unknown com.apple.launchd[1] <Warning>: (UIKitApplication:rdio[0x8d67]) Spawned and waiting for the debugger to attach before continuing...

Apr 22 17:40:48 unknown com.apple.debugserver-64[7463] <Warning>: Got a connection, waiting for debugger instructions for task "(null)".
Apr 22 17:40:48 unknown com.apple.debugserver-64[7456] <Warning>: 1 [1d20/0303]: error: ::read ( 4, 0x2fee74b0, 1024 ) => 0 err = Bad file descriptor (0x00000009)
Apr 22 17:40:48 unknown mobile_installation_proxy[7459] <Error>: main: Could not receive request from host.
Apr 22 17:40:48 unknown mobile_installation_proxy[7453] <Error>: main: Could not receive request from host.
Apr 22 17:40:49 unknown kernel[0] <Debug>: launchd[7464] Builtin profile: container (sandbox)
Apr 22 17:40:49 unknown kernel[0] <Debug>: launchd[7464] Container: /private/var/mobile/Applications/ACD6B3DD-DD6D-4A20-8532-03274A8431DF [69] (sandbox)

(the guid is definitely the app's)
Comment 1 Chris Toshok 2012-04-24 00:06:05 UTC
happened 5 times in the past 1:25.
Comment 2 Chris Toshok 2012-04-24 00:06:18 UTC
also, happens both with wifi debugging and usb
Comment 3 Rolf Bjarne Kvinge [MSFT] 2012-04-30 07:20:07 UTC
I've tried to reproduce this a lot, and I've only seen it happen once :|

Is there any data passing from the app to MonoDevelop when this happens (such as Console.WriteLines for instance)? Could you add periodic CWLs to see if that makes it better? I think it might be the connection that somehow hangs if nothing is being transferred for a while, but given how seldom it happens to me I can't confirm anything.
Comment 4 Matt Crocker 2012-05-08 18:31:04 UTC
Created attachment 1829 [details]
test case

This seems to happen more often when you are redeploying an app, and more reliably with the test app  attached. I'm not sure why, but the network request threads seem to be important. With the threads enabled the app hangs within 30 seconds of redeploying. With the threads disabled redeploying works fine.

Steps to reproduce with Monotouch 5.2.11 and iOS 5:
1. Build the attached sample app in debug mode
2. Deploy the app to a device
4. Start debugging
5. Note that the clock label is ticking as expected
6. Click stop in MD (or press the lock button on the phone, as that seems to cause a disconnect)
7. Note that the app appears to shut down.
8. Start debugging again
8. Note that it starts fine and appears to work
10. Note that the clock will stop within 10 seconds to a minute
Comment 5 Rolf Bjarne Kvinge [MSFT] 2012-05-09 10:02:03 UTC
Created attachment 1834 [details]
stacktrace showing deadlock
Comment 6 Rolf Bjarne Kvinge [MSFT] 2012-05-09 10:05:18 UTC
Zoltan, the stack trace in comment #5 shows a deadlock inside the aot-runtime while debugging, can you have a look? I can get the full crash report too if you need it.
Comment 7 Rolf Bjarne Kvinge [MSFT] 2012-05-09 10:24:02 UTC
Created attachment 1835 [details]
another deadlock stacktrace
Comment 8 Rolf Bjarne Kvinge [MSFT] 2012-05-09 10:28:09 UTC
Adding the following code to a project will create useful stack traces in Xcode for these hangs:

[System.Runtime.InteropServices.DllImport ("libc")]
static extern int strlen (IntPtr zero);
static unsafe void Tick ()
    int zero = 0;
    int* z = &zero;
    // This is so that strlen is initialized (so no/fewer locks are required later)
    strlen ((IntPtr) z); 
    System.Threading.Thread.Sleep (30000);
    strlen (IntPtr.Zero); // Create a crash report.

and then just add this as the first line in Main:

    new System.Threading.Thread (Tick).Start ();
Comment 9 Zoltan Varga 2012-05-09 14:41:45 UTC
This is the same as #3411, which is supposed to be fixed by ed2ae5ee26a601855825ae68a3e8205ea7fbfa49.
Comment 10 Rolf Bjarne Kvinge [MSFT] 2012-05-10 05:34:52 UTC
I am running with ed2ae5ee26a601855825ae68a3e8205ea7fbfa49, so it looks like it's not quite fixed by that commit.
Comment 11 Zoltan Varga 2012-05-10 08:35:37 UTC
That patch can only work if 'soft-debug' is passed to the AOT compiler, otherwise the AOT compiler will generate direct calls between methods, which prevents the AOT runtime from registering the called methods into the JIT info tables. mtouch doesn't pass that flag for system assemblies.
Comment 12 Zoltan Varga 2012-05-10 08:52:32 UTC
So either mtouch needs to pass the 'soft-debug' flag for all assemblies, or we need to add another flag like 'no-direct-calls' and pass that when compiling in debug mode.
Comment 13 Zoltan Varga 2012-05-10 09:51:06 UTC
Should be fixed in mt master and mobile-master.
Comment 14 Rolf Bjarne Kvinge [MSFT] 2012-05-10 11:11:52 UTC
I can confirm it looks fixed, I can't reproduce the hang anymore with the test case from comment #4.
Comment 15 Rolf Bjarne Kvinge [MSFT] 2012-05-25 18:23:25 UTC
*** Bug 5333 has been marked as a duplicate of this bug. ***
Comment 16 Matt Crocker 2012-07-05 19:09:10 UTC
Hey, I heard this fix was in the 5.3.4 beta channel build, but when I switch to that version I can no longer debug on device at all.

I removed the app from the device, rebuild in Device/Debug mode, deployed to device, then pressed debug. The console then says:
Please ensure your device is connected...
Connected to: matt's phone
Launching /private/var/mobile/Applications/82982B5E-0117-4B45-8211-66A5AAA8538E/MojoPriest.app -debugtrack -monodevelop-port 10000 -connection-mode usb

followed by a dialog say "Could not connect to debugger"

I'm running the following:
XCode Version 4.3.2
Monotouch 5.3.4
Mac OS X 10.7.3
iPhone 4S with iOS 5.0.1

Any tips?
Comment 17 Rolf Bjarne Kvinge [MSFT] 2012-07-06 05:05:06 UTC
Matt, what's printed to the ios device log when this happens?
Comment 18 Matt Crocker 2012-07-06 20:26:38 UTC
When I try to debug, the device console shows this: https://pastee.org/x7asb
I assume the important bit is "Failed to load AOT module 'mscorlib' while running in aot-only mode: not compiled with --aot=full." when I launch the debug version by tapping the icon.

In case it is helpful here is the full output from the build process: https://pastee.org/jgmhs

Comparing build output between the two versions it looks like there's a comma missing?



Note missing comma in "ntrampolines=2048,nimt-trampolines=512"
Comment 19 Miguel de Icaza [MSFT] 2012-07-07 11:56:42 UTC
Small update:

I believe that adding a comma to at the end of the extra arguments to aot will fix that on this release, this was reported previoysly here:


CHange your --aot= to have a trailing comma
Comment 21 Zoltan Varga 2012-07-09 14:48:52 UTC
Please do.