|Summary:||Xamarin.Android projects hit the multidex limit much quicker than Android Studio projects|
|Product:||Android||Reporter:||Jon Douglas [MSFT] <jon.douglas>|
|Severity:||normal||CC:||chris.bramley, christophe.oosterlynck, cody.beyer, d.weber, dahaligas, damian, david.ortinau, dhaligas, jon.dick, malcolm.jack, mono-bugs+monodroid, pavel.fedotovsky, peter, pragma.mobilexp, shane94, tomasz, yves_delcoigne|
|Tags:||Is this bug a regression?:||---|
|Last known good build:|
Description Jon Douglas [MSFT] 2017-04-12 18:50:31 UTC
*Description: I've been investigating a potential issue with regards to the multidex limit in Xamarin.Android projects versus Android Studio projects. What I've noticed is that Android Studio projects seem to handle many more library references than the Xamarin.Android counterpart. It seems that the `CompileToDalvik` MSBuild task that invokes dx.jar causes Xamarin.Android projects to reach the multidex limit faster than the Android Studio counterpart. *Reproduction: For my samples, I took a blank Xamarin.Android project and a blank Android Studio project. I then added as many Android Support Libraries as I could until I hit the dex limit. Here is what I found out: 1. Xamarin.Android was only able to hold 24 references to support libraries before hitting the dex limit(Libraries have different dex counts, but using this number as reference to Android Studio): <?xml version="1.0" encoding="utf-8"?> <packages> <package id="Xamarin.Android.Support.Animated.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Compat" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Core.UI" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Core.Utils" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.CustomTabs" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Design" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Exif" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Fragment" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Media.Compat" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Percent" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Recommendation" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Transition" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v13" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v14.Preference" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v4" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.AppCompat" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.CardView" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.GridLayout" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.MediaRouter" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.Palette" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.Preference" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.v7.RecyclerView" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Android.Support.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" /> <package id="Xamarin.Build.Download" version="0.4.3" targetFramework="monoandroid71" /> </packages> With these packages all added, dx.jar will fail stating that multidex needs to be enabled: https://gist.github.com/JonDouglas/73a2562e5a658ed41885904e3385c564 Thus it claims that 67288 references are being used here from the output of dx.jar. The other strange observation here is that some of these libraries that have very different counts end up becoming the same number. You will see a repeated "2422" as a count for many libraries when the actual .jar differs. 2. Android Studio had no problem with the same packages, and it didn't even get close to the dex limit: compile 'com.android.support:support-compat:25.3.1' compile 'com.android.support:support-core-ui:25.3.1' compile 'com.android.support:support-core-utils:25.3.1' compile 'com.android.support:customtabs:25.3.1' compile 'com.android.support:design:25.3.1' compile 'com.android.support:exifinterface:25.3.1' compile 'com.android.support:support-fragment:25.3.1' compile 'com.android.support:percent:25.3.1' compile 'com.android.support:support-media-compat:25.3.1' compile 'com.android.support:support-v4:25.3.1' compile 'com.android.support:appcompat-v7:25.3.1' compile 'com.android.support:cardview-v7:25.3.1' compile 'com.android.support:gridlayout-v7:25.3.1' compile 'com.android.support:mediarouter-v7:25.3.1' compile 'com.android.support:palette-v7:25.3.1' compile 'com.android.support:recyclerview-v7:25.3.1' compile 'com.android.support:preference-v7:25.3.1' compile 'com.android.support:support-v13:25.3.1' compile 'com.android.support:preference-v14:25.3.1' compile 'com.android.support:support-annotations:25.3.1' compile 'com.android.support:leanback-v17:25.3.1' compile 'com.android.support:recommendation:25.3.1' compile 'com.android.support:preference-leanback-v17:25.3.1' compile 'com.android.support:support-media-compat:25.3.1' compile 'com.android.support:support-annotations:25.3.1' `classes.dex` count: 25966 *Investigation: However I then investigated each individual `classes.jar` being inputted in the dx.jar command from the gist above and noticed that the counts weren't even close. https://gist.github.com/JonDouglas/e576a023cfd7f014035e86b0945b3936 This gist shows an output of 21696. The other fact is that if you remove a couple libraries from NuGet for the Xamarin.Android application and successfully rebuild, the dex count is still around the ~22k count. This isn't even close to the 65,536 limit and thus I suspect something is going on. I haven't been able to figure out what that is quite yet. I even went as far as adding --verbose logging to dx.jar and seeing if it was adding certain libraries multiple times. https://gist.github.com/JonDouglas/2b5eb5fc3bb2a4a390cf879be1aedbe8 However I did not see anything that quite caught my eye here. My overall suspicions: 1) Something is being added to dx.jar many times to the point of hitting the dex limit 2) Optimization/pre-dexing is different in Android Studio vs. Xamarin.Android 3) The strange number of "2422" for libraries that have different counts seems suspicious *Version Information: Microsoft Visual Studio Enterprise 2017 Version 15.1 (26403.0) Release VisualStudio.15.Release/15.1.0+26403.0 Microsoft .NET Framework Version 4.6.01586 Installed Version: Enterprise Xamarin 18.104.22.168 (3f99c5a) Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android. Xamarin.Android SDK 22.214.171.124 (b16fb82) Xamarin.Android Reference Assemblies and MSBuild support. Xamarin.iOS and Xamarin.Mac SDK 10.8.0.174 (7656cc6) Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
Comment 2 Peter Major 2017-04-19 10:36:20 UTC
I'm also seeing this. It's impossible to update from AppCompat 23.3.0 to 25.1.1 or 25.3.x without hitting the limit. This seems like a massive problem! The number of packages with counts exactly the same is very suspicious. My guess is that it's including the count for the dependencies (i.e. for every package using 'android.support.compat', the package's method count includes the methods in 'android.support.compat') trouble writing output: Too many field references: 78376; max is 65536. You may try using --multi-dex option. References by package: 3 android.accounts 30 android.app 1 android.content 37 android.content.pm 10 android.content.res 3 android.database 46 android.graphics 1 android.graphics.drawable 2 android.media 1 android.media.browse 2 android.net 25 android.os 3 android.print 1 android.provider 8 android.runtime 1 android.support.annotation 2656 android.support.compat 2656 android.support.coreui 2656 android.support.coreutils 2760 android.support.customtabs 2656 android.support.design 107 android.support.design.internal 700 android.support.design.widget 2656 android.support.fragment 2815 android.support.graphics.drawable 2656 android.support.graphics.drawable.animated 2656 android.support.mediacompat 2835 android.support.transition 2656 android.support.v4 13 android.support.v4.accessibilityservice 17 android.support.v4.animation 885 android.support.v4.app 114 android.support.v4.content 1 android.support.v4.content.pm 1 android.support.v4.content.res 11 android.support.v4.graphics 41 android.support.v4.graphics.drawable 4 android.support.v4.hardware.display 12 android.support.v4.hardware.fingerprint 10 android.support.v4.internal.view 388 android.support.v4.media 379 android.support.v4.media.session 10 android.support.v4.net 19 android.support.v4.os 63 android.support.v4.print 10 android.support.v4.provider 55 android.support.v4.text 6 android.support.v4.text.util 118 android.support.v4.util 434 android.support.v4.view 138 android.support.v4.view.accessibility 8 android.support.v4.view.animation 495 android.support.v4.widget 605 android.support.v7.app 2656 android.support.v7.appcompat 2656 android.support.v7.cardview 8 android.support.v7.content.res 88 android.support.v7.graphics 18 android.support.v7.graphics.drawable 437 android.support.v7.media 2656 android.support.v7.mediarouter 2656 android.support.v7.palette 2656 android.support.v7.recyclerview 1 android.support.v7.text 2 android.support.v7.transition 124 android.support.v7.util 77 android.support.v7.view 249 android.support.v7.view.menu 1483 android.support.v7.widget 89 android.support.v7.widget.helper 1 android.support.v7.widget.util 6 android.text 1 android.text.util 2 android.transition 14 android.util 19 android.view 8 android.view.accessibility 1 android.view.inputmethod 15 android.widget 2650 com.google.android.gms 20 com.google.android.gms.actions 13 com.google.android.gms.ads.identifier 83 com.google.android.gms.appdatasearch 2676 com.google.android.gms.appindexing 53 com.google.android.gms.auth 7 com.google.android.gms.auth.api.credentials 57 com.google.android.gms.auth.api.signin 13 com.google.android.gms.auth.api.signin.internal 75 com.google.android.gms.auth.firstparty.shared 2650 com.google.android.gms.base 38 com.google.android.gms.clearcut 94 com.google.android.gms.common 87 com.google.android.gms.common.api 179 com.google.android.gms.common.api.internal 42 com.google.android.gms.common.data 49 com.google.android.gms.common.images 171 com.google.android.gms.common.internal 1 com.google.android.gms.common.internal.safeparcel 4 com.google.android.gms.common.server 12 com.google.android.gms.common.server.converter 33 com.google.android.gms.common.server.response 62 com.google.android.gms.common.stats 27 com.google.android.gms.dynamic 2 com.google.android.gms.dynamite.descriptors.com.google.android.gms.flags 16 com.google.android.gms.flags.impl 2753 com.google.android.gms.gcm 47 com.google.android.gms.iid 372 com.google.android.gms.internal 2666 com.google.android.gms.measurement 350 com.google.android.gms.measurement.internal 32 com.google.android.gms.playlog.internal 14 com.google.android.gms.search 6 com.google.android.gms.security 26 com.google.android.gms.signin.internal 2650 com.pelicanexchange 2799 com.roughike.bottombar 2893 com.theartofdev.edmodo.cropper 1 com.xamarin.forms.platform.android 2 default 4 java.lang 7 java.lang.annotation 1 java.nio 6 java.util 3 java.util.concurrent 1 javax.microedition.khronos.egl 2 md507ae2fe0c5e6c7f1cce2073326250bfa 2 md50bd63072d32fd72e10115bcaf84d340f 36 md5270abb39e60627f0f200893b490a1ade 8 md531f8cc8d4d39b2fa02686ee0eb368905 4 md534e018afb580d217ff06dd0dbae3d572 4 md5539939fcd5322368b18e9187b5e77d3d 14 md56987f6cc1cad2bad7d1513f02edd2700 54 md5716b2bf943f46e4067e009323bbbb898 6 md584f84b531215f3b805721fa27da129c9 4 md59f70a99687498e7ba187118950981d26 12 md5a104545e4d19c4ffe9ec3d5074a3b979 6 md5a4ed55898dc7fb7b87f5830ba08179d9 2 md5afd03715c12d7cea439a2ca4d0e82348 160 md5b60ffeb829f638581ab2bb9b1a7f4f3f 34 md5c25e555921ab1c976dcce693fa7017e6 2 md5cf6f91d9bcbb245b12b8f240e1ec031d 30 md5f9b7691a3024e3da489938deb6fbfb67 6 mono 1 mono.android 4 mono.android.accessibilityservice 2 mono.android.accounts 12 mono.android.animation 31 mono.android.app 2 mono.android.bluetooth 20 mono.android.content 2 mono.android.database.sqlite 6 mono.android.drm 6 mono.android.gesture 2 mono.android.graphics 2 mono.android.graphics.drawable 8 mono.android.hardware 2 mono.android.hardware.display 2 mono.android.hardware.input 2 mono.android.inputmethodservice 8 mono.android.location 60 mono.android.media 16 mono.android.media.audiofx 2 mono.android.media.effect 2 mono.android.media.midi 2 mono.android.media.session 2 mono.android.media.tv 2 mono.android.net 6 mono.android.net.nsd 2 mono.android.net.sip 18 mono.android.net.wifi.p2p 2 mono.android.nfc 10 mono.android.os 10 mono.android.preference 2 mono.android.renderscript 6 mono.android.runtime 6 mono.android.sax 2 mono.android.speech 4 mono.android.speech.tts 12 mono.android.support.design.widget 2 mono.android.support.transition 4 mono.android.support.v4.app 4 mono.android.support.v4.content 2 mono.android.support.v4.media.session 2 mono.android.support.v4.os 18 mono.android.support.v4.view 4 mono.android.support.v4.view.accessibility 12 mono.android.support.v4.widget 6 mono.android.support.v7.app 2 mono.android.support.v7.graphics 2 mono.android.support.v7.media 32 mono.android.support.v7.widget 2 mono.android.text 2 mono.android.transition 72 mono.android.view 4 mono.android.view.accessibility 2 mono.android.view.animation 2 mono.android.view.textservice 8 mono.android.webkit 72 mono.android.widget 2 mono.com.google.android.gms.common.api 2 mono.com.google.android.gms.common.images 2 mono.com.google.android.gms.security 4 mono.com.roughike.bottombar 8 mono.com.theartofdev.edmodo.cropper 4 mono.java.lang 2 mono.java.util 2 mono.javax.xml.transform 2 mono.net.hockeyapp.android 2839 net.hockeyapp.android 2 net.hockeyapp.android.adapters 65 net.hockeyapp.android.metrics 82 net.hockeyapp.android.metrics.model 79 net.hockeyapp.android.objects 80 net.hockeyapp.android.tasks 86 net.hockeyapp.android.utils 39 net.hockeyapp.android.views 2 opentk 2 opentk.platform.android 2 opentk_1_0 2 opentk_1_0.platform.android 2 pelican.ui.droid.controls 2 pelican.ui.droid.renderers
Comment 3 dhaligas 2017-05-24 19:40:47 UTC
we ran into this as well. hoping xamarin can shed some light on this
Comment 4 dhaligas 2017-05-30 14:48:09 UTC
@jonp any updates on this?
Comment 5 Jon Dick 2017-06-01 14:21:13 UTC
I have an app that I can easily reproduce this in, what's interesting, is as soon as I run proguard, I now seem to get a dex file with a ~30,000 count, no multidex required. If I turn proguard off, I get the error. I don't believe this was the case on previous versions of Xamarin.Android, perhaps it's worth retesting on current stable?
Comment 6 Jon Douglas [MSFT] 2017-06-01 20:39:07 UTC
Although we provide a Proguard target after the Javac task and before the CompileToDalvik task, it is an "opt-in" situation meaning that one has to enable Proguard for the desired behavior. Android Studio doesn't seem to do this however. They have a default of minifyEnabled = false for projects unless flipped. Thus I suspect that Proguard is still running on their end between the two tools(javac -> dx.jar or inside jack) as the legacy Android Build Process is more or less the following: Source Code / Library Code -> javac -> Java bytecode (.class) -> proguard -> minimized bytecode (.class) -> dex -> DEX bytecode (.dex) Thus a form of optimization via shrinking is happening on the Android Studio side and not on our side. I believe this should happen by default for library references like Google Play Services where they provide their own proguard configuration already. That ontop of the default proguard-android.txt, we should also give the option of specifying the further optimized proguard-android-optimize.txt if a customer wanted to try that inside their project for further optimization. We allow a custom proguard config for further control as well if it's needed past one of the two configurations noted above. So there are two conditions for Proguard to run in a Debug configuration: #1 You have to disable the shared runtime #2 The linker must run (Thus it cannot be set to None) My original attempt at documenting this feature shows that only a Release configuration can run the Proguard Task because the conditions are easier to figure out: https://developer.xamarin.com/guides/android/deployment,_testing,_and_metrics/proguard/#using Thus for this use-case, one would have to enable Proguard in a Debug configuration with the conditions noted above for Proguard to properly shrink all of the .class items before sending off to dx.jar. My testing with this same project gave a total count of 21k after running proguard and dx.jar. This is approximately the same that I found in Android Studio. I believe some discussion on what we can do to improve this experience needs to happen. Currently the experience of adding too many references in the current Debug configuration is troublesome because it requires the developer to figure out what "java.exe exited with code 2" means, and the error says to turn on multidex when in reality if we ran Proguard by default, we can avoid multidex completely in these use-cases. The foreseeable problems this can cause is unknown to me as I'm looking for some insight in that area. Also if it's possible to run Proguard with the Shared Runtime and Fast Dev, that would be another avenue to explore for Debug configuration. However it seems that $(_ProguardProjectConfiguration) doesn't populate with it enabled.
Comment 7 pragma.mobilexp 2017-06-29 12:48:27 UTC
Any progress being made on this? We are running into the same issues with Xamarin hitting the dex limit way before the same kind of project using Android Studio. This is causing a lot of churn and wasted productivity for us.
Comment 8 Jon Dick 2017-06-29 13:12:48 UTC
In my experience the current work around to avoiding requiring multidex being enabled is to enable Proguard in your project (multidex actually does this implicitly anyway) and have that step working properly. In my one app, I was able to reduce the dex count by just using proguard to an acceptable limit where I otherwise was hitting the dex limit error. We still need to investigate how to handle this in the Xamarin.Android chain (do we implicitly use proguard in the build process? how does that work for proguard config errors and warnings, etc..).
Comment 9 Jon Douglas [MSFT] 2017-07-05 18:17:37 UTC
I am confirming this behavior based on my comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6 and comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c8
Comment 10 dhaligas 2017-07-06 23:55:10 UTC
anyone have a fix for this? it is causing tons of wasted time and messing with the ability to hit breakpoints
Comment 11 Tomasz Cielecki 2017-08-07 07:27:32 UTC
I've also had issues with this recently. I had an app which was running fine without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android Support packages and started to hit the Dex limit. No new NuGet packages were introduced, just updating existing ones. If ProGuard is going to be enabled by default, I think it should be in place, that NuGets and Xamarin Components, such as the Play Services ones add their own ProGuard configuration, similar to how it is done in Android Studio and Gradle. In most cases no one should have to mess around with those configs.
Comment 12 Jon Douglas [MSFT] 2017-08-07 16:06:20 UTC
(In reply to Tomasz Cielecki from comment #11) > I've also had issues with this recently. I had an app which was running fine > without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android > Support packages and started to hit the Dex limit. No new NuGet packages > were introduced, just updating existing ones. > > If ProGuard is going to be enabled by default, I think it should be in > place, that NuGets and Xamarin Components, such as the Play Services ones > add their own ProGuard configuration, similar to how it is done in Android > Studio and Gradle. In most cases no one should have to mess around with > those configs. I definitely agree here. There is already work done in this area to help out: https://github.com/xamarin/XamarinComponents/pull/166
Comment 13 dhaligas 2017-08-07 16:08:36 UTC
@jondouglas can this be used now? also when proguard is on by default can we still use fast deployment?
Comment 14 Jon Dick 2017-08-07 16:15:46 UTC
Additionally v25.4.0 and up will ship with proguard configs in the nuget packages and targets files to have them added. Eventually xamarin.android will properly detect them from .aar files too.
Comment 15 Jon Douglas [MSFT] 2017-08-07 16:23:38 UTC
(In reply to dhaligas from comment #13) > @jondouglas can this be used now? also when proguard is on by default can > we still use fast deployment? You can use this task as long as your Xamarin.Build.Download is 0.4.6 and above as it will contain the XamarinBuildAndroidAarProguardConfigs Task. See what Jon Dick has to say regarding shipping proguard configs in the NuGet packages as I believe that's the missing element in the current NuGets. You will see a "Proguard" folder with a proguard.txt if the library includes it's own configuration. (A quick example is the Xamarin.Android.Support.Design package 25.4.0 - https://www.nuget.org/packages/Xamarin.Android.Support.Design/25.4.0-rc1) Unfortunately Proguard in a Debug configuration still requires the Shared Runtime to be turned off and the Linker to run to work. (Explained here: https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6)
Comment 16 dhaligas 2017-08-09 16:35:17 UTC
@jondick just updated to the latest support libs and now I cannot build https://forums.xamarin.com/discussion/98777/invalidprojectfileexception-cycle-in-target-dependencies-detected-build-failed Error: Error building target _XamarinAndroidBuildAarProguardConfigs: Microsoft.Build.BuildEngine.InvalidProjectFileException: Cycle in target dependencies
Comment 17 Malcolm Jack 2017-08-15 21:03:07 UTC
super disappointed this isn't getting a higher priority, and now pushed out to 15.5 :(
Comment 18 dhaligas 2017-08-15 21:47:44 UTC
I agree with Malcom. I cannit use fast deployment and has doubled my android build times. Very unproductive with this issue
Comment 19 Jon Dick 2017-09-28 14:26:54 UTC
*** Bug 53059 has been marked as a duplicate of this bug. ***
Comment 20 dean.ellis 2018-02-26 10:57:48 UTC
I have added an issue at https://github.com/xamarin/xamarin-android/issues/1337.