Bug 56653 - Zygote crashes
Summary: Zygote crashes
Status: RESOLVED FIXED
Alias: None
Product: Android
Classification: Xamarin
Component: Mono runtime / AOT Compiler (show other bugs)
Version: 7.3 (15.2)
Hardware: PC Windows
: --- normal
Target Milestone: ---
Assignee: Marek Habersack
URL:
: 57394 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-05-19 03:27 UTC by Kent
Modified: 2017-08-11 13:42 UTC (History)
13 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
RESOLVED FIXED

Description Kent 2017-05-19 03:27:30 UTC
I'll be honest: I'm not 100% sure this is a Xamarin issue. But I am pretty certain it must either be mono or Android at fault here, so I'm looking for some guidance in moving forward and resolving this.

We're seeing a bunch of crashes in Google Play Console with stack traces like this:

    java.lang.RuntimeException: 
      at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1358)
    Caused by: java.lang.reflect.InvocationTargetException: 
      at java.lang.reflect.Method.invoke(Native Method:0)
      at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1468)

There isn't any other info provided with the crash apart from this metadata:

    Report timestamp
    Yesterday, 11:12 PM
    Application version
    751
    Android version
    Android 7.0
    Device
    Galaxy S7 (herolte)
    Manufacturer Samsung RAM (MB) 4096 Screen size 1440 × 2560 Screen density (dpi) 640 Native platform armeabi-v7a OpenGL ES version 3.1 CPU make Samsung CPU model Exynos 8890

We have Raygun in our app, but the crash never comes through there. I am told this is because Raygun hasn't even had a chance to initialize yet. Makes sense given that Zygote is involved in bootstrapping the app.

Is there any chance that mono is failing to bootstrap and causing this crash? If not, any ideas what might be causing this/how to diagnose further?
Comment 1 Marek Habersack 2017-06-07 21:43:28 UTC
@Kent, if you don't see any Mono/Xamarin.Android messages in the captured crash then our runtime probably hasn't had the chance to initialize yet. The bad news is that looking at the ZygoteInit source in all the Nougat versions doesn't give us much because the source in all of those versions is not longer than ~900 lines. There is, however, a number of instances when RuntimeException can be thrown from the `main` function (see https://github.com/android/platform_frameworks_base/blob/nougat-mr2.2-release/core/java/com/android/internal/os/ZygoteInit.java#L712). This is, however, just a guessing game since we can't tell what is the actual code doing above. It is, in all likelihood, a version of Android N modified by Samsung.
It does not seem that Xamarin.Android can be running at this point, but we can exclude the possibility that the issue is somewhere in our initialization code.

It would be ideal if you could provide us with a repro, but I don't think it's possible given the data at hand. I'm afraid the only hope to reproduce is would be to run a debug build of your application 
on the device mentioned above and hope that logcat provides more information.
Comment 3 John 2017-06-09 11:57:05 UTC
Hi Marek / Kent

I would just like to chip in and point you at the following forum discussion:

https://forums.xamarin.com/discussion/96443/zygoteinit-errors

It appears from the comments, that this has become a very common issue that looks as if it only really started in May of this year. Unfortunately, my App only launched in May, so I can't confirm if that was when the problem started unfortunately.

In my own case, I have two separate (similar) sets of occurrences. The first only happens on Android 6.0, 7.0, and 7.1:

java.lang.RuntimeException: 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:762)
Caused by: java.lang.reflect.InvocationTargetException: 
at java.lang.reflect.Method.invoke(Native Method:0)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:872)

The second only happens on Android 5.0 and 5.1:

java.lang.RuntimeException: 
at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:815)
Caused by: java.lang.reflect.InvocationTargetException: 
at java.lang.reflect.Method.invoke(Native Method:0)
at java.lang.reflect.Method.invoke(Method.java:372)
at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1020)

I don't know if the slight difference in stack traces might shed any light?
This is on a wide variety of hardware. I have seen it once myself (LG G4 Android 6.0), when the app closed immediately on launching, but then works fine after that.
Comment 4 Marek Habersack 2017-06-09 12:21:37 UTC
@John, thanks for the info! The traces you provided point to code we can hope to find in one of the public Android branches and hope that it sheds some light on the issue.
Comment 5 John 2017-06-11 09:53:29 UTC
Marek,

I may have a little more information. I implemented a hidden "throw a divide by zero" option in my app a while ago when I was testing my internal log file generation. This happens when a user types in a specific value into a text box then presses a particular button.

On 17 May, the Google Play Developer console changed. You can now view error reports which are submitted automatically (if the user opts-in to send usage data and diagnostics), separately from those where the user explicitly submits an error report. Here are the interesting results:

If I opt-out, and send an explicit report, this is what I see in the Google Developer console:

java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:620)
Caused by: java.lang.reflect.InvocationTargetException
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:730)
	... 1 more
Caused by: android.runtime.JavaProxyThrowable: System.Exception: Trapped exception in DownloadPropertiesPage.OnDownload: Attempted to divide by zero.
  at Jbct.AnyMap.Guard+<LogOnExceptionAsync>d__3.MoveNext () [0x000ca] in <cd46ae600a494c8cbf38c66a0d73c275>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () [0x00000] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at Jbct.AnyMap.DownloadPropertiesPage+<OnDownload>d__8.MoveNext () [0x0007d] in <cd46ae600a494c8cbf38c66a0d73c275>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>m__0 (System.Object state) [0x00000] in <0b312dfa67a84d4dbdce6f4cc3623c73>:0 
  at Android.App.SyncContext+<Post>c__AnonStorey0.<>m__0 () [0x00000] in <7e5ff0a6057f44a99e00656c79ab1e0e>:0 
  at Java.Lang.Thread+RunnableImplementor.Run () [0x0000b] in <7e5ff0a6057f44a99e00656c79ab1e0e>:0 
  at Java.Lang.IRunnableInvoker.n_Run (System.IntPtr jnienv, System.IntPtr native__this) [0x00009] in <7e5ff0a6057f44a99e00656c79ab1e0e>:0 
  at (wrapper dynamic-method) System.Object:da1d8b14-6e9d-410a-aac6-18d5576cd594 (intptr,intptr)
	at mono.java.lang.RunnableImplementor.n_run(Native Method)
	at mono.java.lang.RunnableImplementor.run(RunnableImplementor.java:30)
	at android.os.Handler.handleCallback(Handler.java:739)
	at android.os.Handler.dispatchMessage(Handler.java:95)
	at android.os.Looper.loop(Looper.java:148)
	at android.app.ActivityThread.main(ActivityThread.java:5525)
	... 3 more

Pretty much as I would expect - you can see the "Divide by zero". However, if I opt-out, then this is all that is reported in the Developer console:

java.lang.RuntimeException: 
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:620)
Caused by: java.lang.reflect.InvocationTargetException: 
  at java.lang.reflect.Method.invoke(Native Method:0)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:730)

Two things: (a) the useful stack trace is missing and (b) the error report includes ZygoteInit, which seems to usually be interpreted and explained as an error on startup. This is nothing to do with startup in my test case, as the exception is thrown when the user clicks on a button.

I can't believe that all automatic error reporting to all Android developers has now become useless (or maybe it has!). I would think that maybe it is something in the way that Xamarin exceptions are processed which is causing the key stack trace to be omitted?

To just confirm: The Google Play console was changed on 17th May, which explains why these problems have only recently been reported.

Hope this helps.
John
Comment 6 John 2017-06-11 12:40:53 UTC
Typo: "However, if I opt-out, then this is all that is reported in the Developer console:" should read "However, if I opt-IN, then this is all that is reported in the Developer console:" !
Comment 7 urs_ammann 2017-06-11 18:32:31 UTC
With the new error reporting system from google the second part of the call stack is absent and the most errors appeare with this no information like this:

java.lang.RuntimeException: 
  at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:620)
Caused by: java.lang.reflect.InvocationTargetException: 
  at java.lang.reflect.Method.invoke(Native Method:0)
  at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:730)

It should be easy to upload a test app to the google store to verify this.
Comment 8 John 2017-06-11 19:15:03 UTC
In terms of Xamarin, I have already verified it with my own app as detailed earlier: Automatically submitted error reports definitely omit the second (useful) part of the call stack.

The question is: Is this true of all Android apps, or just those developed with Xamarin? That's not something I can verify myself unfortunately, as I don't have a native Java app to test with.

If it's just Xamarin, then what is it that causes the second part to be omitted, and can it be fixed?
Comment 9 Marek Habersack 2017-06-12 13:57:50 UTC
I don't *know* the answer to the whether it's a Xamarin issue question, but my instinctive reaction was that perhaps the automated reporting parses the app's output looking only for Java exceptions that are caught and reported by Zygote. On the other hand, however, they aren't capturing the Java  JavaProxyThrowable exception Xamarin runtime throws. There's no telling at this point what's really going on. However, thank you @John for the info - it gives us at least some clue as to which way we should go to investigate the issue.
Comment 10 Marek Habersack 2017-06-12 14:56:40 UTC
@peterc, can you set up the test as suggested in comment 7? Thanks!
Comment 11 Marek Habersack 2017-06-12 18:29:05 UTC
*** Bug 57394 has been marked as a duplicate of this bug. ***
Comment 12 amantha@amantha.cz 2017-06-13 17:26:04 UTC
Just after cleaning and rebuilding solution, I deployed it again to Google Play, and it's working now.
Comment 13 John 2017-06-14 08:53:13 UTC
Marek / Peter,

Have you made any progress on this? I updated to all the latest Xamarin components yesterday (I was a little out of date) and redeployed to Google Play, and am seeing exactly the same errors with no Xamarin stack trace (which I did still expect to be honest).

I am in the unfortunate position of having launched my first Xamarin Forms app just over a month ago. I am getting a huge number of errors given the small number of installs, but have absolutely nothing to go on in terms of finding the cause, since I have no error reports which pre-date the change to the Google Play developer console, so any help / suggestions appreciated!!

It does make sense that there may be some form of string filtering going on within the console, albeit only for automatically submitted crash reports for some reason.
Comment 14 Marek Habersack 2017-06-14 13:01:46 UTC
@John, are you able to repro the crash on one of your devices? If not, perhaps we could help in this regard - if we have a device available we could run your application triggering the crash while you're opted out of the automatic reports, which might give us something to work with. As it currently stands, this crash can be anything and we have no starting point :(

Before rebuilding, did you thoroughly clean your project? And by that I mean removing all the `bin`, `obj` directories as the `.vs` directory in the top-level dir of the project.

Also, I don't know if you use TestCloud (https://www.xamarin.com/test-cloud) but it could help as it would make it possible to run the app on many devices and if you used a different app id then there would be no reason to opt-out of the automatic crash reporting - we should see the crash with all the traces etc.
Comment 15 John 2017-06-14 17:25:11 UTC
OK. Deep breath. Long post...

There are basically two issues I suppose:
1. My app is crashing and I don't know why. That I'm less concerned about at the moment - explained below
2. Google Play developer console is missing the Xamarin stack traces

So, focusing on 2:

It seems there was another Xamarin update in the last day or two, so I updated all my components again, and re-deployed to Google Play. For reference, my versions are:

Xamarin 4.5.0.486
Xamarin.Android 7.3.1.2
Xamarin.Forms 2.3.4.247

The build process was: (1) quit VS; (2) erase all obj, bin, and .vs folders as suggested; (3) launch VS, build and archive immediately.

There is no change in behaviour. To recap - I am forcing a divide by zero exception in my app with a particular input sequence on one of the pages. Here's possibly a more comprehensive (and accurate) description of what I am doing, and what is going on:

The developer console crash reports are split into New ANRs and Previous ANRs.
According to Google, Previous ANRs: "... shows all ANRs & crashes collected from Android devices whose users have reported individual crash and ANR events".
New ANRs:  "... as of May 17th 2017, the new section shows all ANRs & crashes collected from Android devices whose users have opted in to automatically share usage and diagnostics data and has a higher volume of reports".

Whether I choose to share usage and diagnostics data (LG G4, Android 6.0) or not has absolutely no effect on the behaviour, which is:

As soon as the app crashes, a dialog "app name has stopped" pops up, with two options: "Report", and "OK". If I choose "Report", then the crash report appears in the Previous ANRs section with a full Xamarin stack trace, and if I choose "OK", then it appears in the New ANRs section with just the ZygoteInit part of the stack trace. I can only assume that the vast majority of users choose "OK" (or more likely never get the option), as the New ANR crashes outweigh the Previous ANR crashes by about 100:1.

What I have since done is implement HockeyApp and re-deployed (again!) Again the behaviour is identical regardless of whether I choose to share usage and diagnostics data or not, which is:

As soon as the app crashes, it closes immediately but with no "app name has stopped" dialog being shown. Next time the app launches there is an option to send a report or to not send a report. If I send the report, then it is sent to the HockeyApp dashboard with a full stack trace, and to the Google Play New ANRs section with a limited ZygoteInit-part-only stack trace. If I don't send the report, then it is sent to neither.

I will let things run for a few days like this, and see how and where real-World crashes are being reported.
Comment 16 urs_ammann 2017-06-14 19:40:31 UTC
I have upload a test app with the name ZygoteCrashes. This App is in the Beta State in the Google App Store and only accessible for me.
The App has two buttons, the first throws an error "div by zero" the second a NullReferenceException exception.
I have test with two devices Android 5 (Galaxy S4) (No update on Android 5) and Android 7 (Moto G 5th gen)
After click on a button, the App crashes and a messagebox appears with the text "Unfortunately ZygoteCrashes has stopped" with buttons "Report" and "Ok".
I tried both the "Ok" and the "Report"

As i awaited in the old system i received the full call stack and in the new system only the useless Zygote call stack. 
Android 5 did not send to the new error system.
The errors was sent with a delay of hours to the new system.

I hope we have not to live with this no information call stack. I have a lot of crashes with this...
Comment 17 Peter Collins 2017-06-14 21:47:16 UTC
I've managed to reproduce this scenario using a very simple app which also throws a DivideByZeroException (similar to Comment 15). I've since forced this crash on a few devices, and I'm seeing the truncated exception stack trace in the 'new' section of the Google Play Console as reported. I can also confirm that clicking on the 'Report' option which is present in the crash dialog on device will send a more detailed crash log to the 'old' crash reporting section of the console.

@John you've mentioned that you've reached out to Google on this, do you have an open ticket somewhere that we can reference?
Comment 18 John 2017-06-15 06:18:21 UTC
I raised it with Google by email. The ticket reference is 3-4373000017801.

I received a response yesterday. I haven't replied, as I suspect from the phrasing, it isn't going to go anywhere! Response below:

"Thank you for contacting Google Play Developer Support.
 
I'm sorry you are experiencing issues with the way crashes are reported in the Play Console and thank you for reporting this case.
 
We are constantly working to improve our crash reporting to provide more information to help developers diagnose issues.
 
We’re constantly adding new features and functionality, so please stay tuned. Make sure to check the Android Developers Blog for new features and updates:
http://android-developers.blogspot.com/
 
Also, you can find Developer Support communities listed here: 
http://developer.android.com/support.html
 
If you have other questions about using the Google Play Console, please let me know and I’ll be happy to help.
 
Thanks for your patience as we work to improve Google Play!"
Comment 19 Marek Habersack 2017-06-16 09:20:37 UTC
Currently, this appears to be an issue with Google crash reporting system (possibly just the console?) as our behavior hasn't changed in this regard. We'll try to contact Google to see what can be done about it, as I don't see anything we can change on our side at this point. I'll update this bug once we have any information. Thanks for all the reports and your patience guys!
Comment 20 urs_ammann 2017-06-17 15:04:19 UTC
https://issuetracker.google.com/issues/62738358
Comment 21 Jonathan Pryor 2017-06-20 20:32:14 UTC
Peter Collins did some additional testing with Java apps:

https://gist.github.com/pjcollins/ebb4da36945fa253cc3049b6275d619c

At present, it looks like Zygote and company are special-casing java.lang.Exception and java.lang.Error behavior. JavaProxyThrowable instead inherits from java.lang.Throwable -- the base class of Error and Exception -- and Zygote/etc. don't appear to support this.

We should alter JavaProxyThrowable to instead inherit from java.lang.Error:

https://github.com/xamarin/xamarin-android/blob/5777337/src/Mono.Android/Android.Runtime/JavaProxyThrowable.cs

https://github.com/xamarin/java.interop/blob/master/src/Java.Interop/java/com/xamarin/java_interop/internal/JavaProxyThrowable.java#L8
Comment 22 Marek Habersack 2017-06-22 15:18:19 UTC
Potential fixes:

 Xamarin.Android PR: https://github.com/xamarin/xamarin-android/pull/663
 Java.Interop PR: https://github.com/xamarin/java.interop/pull/165
Comment 23 Marek Habersack 2017-06-22 17:08:43 UTC
Fix merged in xamarin-android/master, https://github.com/xamarin/xamarin-android/commit/2b950f15a666f127d89899e280ab0fafddc8cc5c - NOT confirmed that it actually works yet!
Comment 24 Marek Habersack 2017-06-22 17:32:01 UTC
It appears that we've got progress! Peter tested the above XA PR and got these reults: https://gist.github.com/pjcollins/0abc8240b28da4230163e4536127c76b

The exception error message is missing, but the same happens with "pure" Java exceptions, so it must be an issue with Zygote.
Comment 25 John 2017-06-22 18:12:51 UTC
That's great news! And missing the part that mentions Zygote is no bad thing...

I've been using HockeyApp as a stop-gap, and captured and fixed a few exceptions that way, but I'm pretty sure it doesn't catch them all, and the way I've used it (not recommended) to try to catch exceptions on startup (i.e. in OnCreate and not OnResume) I think as actually *causing* exceptions on some devices.

So having a fix to this Zygote issue and going back to the standard Google Console error reporting fix is definitely the way to go (for me anyway).

Thanks for dealing with it so quickly! Any release eta.... ? (the million dollar question)
Comment 26 Marek Habersack 2017-06-23 16:57:40 UTC
@John, it will be part of the upcoming release. Look out for version 7.4.0.11 or newer