I've been investigating a potential issue with regards to the multidex limit in Xamarin.Android projects versus Android Studio projects.
What I've noticed is that Android Studio projects seem to handle many more library references than the Xamarin.Android counterpart.
It seems that the `CompileToDalvik` MSBuild task that invokes dx.jar causes Xamarin.Android projects to reach the multidex limit faster than the Android Studio counterpart.
For my samples, I took a blank Xamarin.Android project and a blank Android Studio project.
I then added as many Android Support Libraries as I could until I hit the dex limit. Here is what I found out:
1. Xamarin.Android was only able to hold 24 references to support libraries before hitting the dex limit(Libraries have different dex counts, but using this number as reference to Android Studio):
<?xml version="1.0" encoding="utf-8"?>
<package id="Xamarin.Android.Support.Animated.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Compat" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Core.UI" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Core.Utils" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.CustomTabs" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Design" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Exif" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Fragment" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Media.Compat" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Percent" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Recommendation" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Transition" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v13" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v14.Preference" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v4" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.AppCompat" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.CardView" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.GridLayout" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.MediaRouter" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.Palette" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.Preference" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.v7.RecyclerView" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Android.Support.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" />
<package id="Xamarin.Build.Download" version="0.4.3" targetFramework="monoandroid71" />
With these packages all added, dx.jar will fail stating that multidex needs to be enabled:
Thus it claims that 67288 references are being used here from the output of dx.jar. The other strange observation here is that some of these libraries that have very different counts end up becoming the same number. You will see a repeated "2422" as a count for many libraries when the actual .jar differs.
2. Android Studio had no problem with the same packages, and it didn't even get close to the dex limit:
`classes.dex` count: 25966
However I then investigated each individual `classes.jar` being inputted in the dx.jar command from the gist above and noticed that the counts weren't even close.
This gist shows an output of 21696.
The other fact is that if you remove a couple libraries from NuGet for the Xamarin.Android application and successfully rebuild, the dex count is still around the ~22k count. This isn't even close to the 65,536 limit and thus I suspect something is going on. I haven't been able to figure out what that is quite yet. I even went as far as adding --verbose logging to dx.jar and seeing if it was adding certain libraries multiple times.
However I did not see anything that quite caught my eye here. My overall suspicions:
1) Something is being added to dx.jar many times to the point of hitting the dex limit
2) Optimization/pre-dexing is different in Android Studio vs. Xamarin.Android
3) The strange number of "2422" for libraries that have different counts seems suspicious
Microsoft Visual Studio Enterprise 2017
Version 15.1 (26403.0) Release
Microsoft .NET Framework
Installed Version: Enterprise
Xamarin 188.8.131.52 (3f99c5a)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.
Xamarin.Android SDK 184.108.40.206 (b16fb82)
Xamarin.Android Reference Assemblies and MSBuild support.
Xamarin.iOS and Xamarin.Mac SDK 10.8.0.174 (7656cc6)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
I'm also seeing this. It's impossible to update from AppCompat 23.3.0 to 25.1.1 or 25.3.x without hitting the limit.
This seems like a massive problem!
The number of packages with counts exactly the same is very suspicious. My guess is that it's including the count for the dependencies (i.e. for every package using 'android.support.compat', the package's method count includes the methods in 'android.support.compat')
trouble writing output: Too many field references: 78376; max is 65536.
You may try using --multi-dex option.
References by package:
we ran into this as well. hoping xamarin can shed some light on this
@jonp any updates on this?
I have an app that I can easily reproduce this in, what's interesting, is as soon as I run proguard, I now seem to get a dex file with a ~30,000 count, no multidex required. If I turn proguard off, I get the error.
I don't believe this was the case on previous versions of Xamarin.Android, perhaps it's worth retesting on current stable?
Although we provide a Proguard target after the Javac task and before the CompileToDalvik task, it is an "opt-in" situation meaning that one has to enable Proguard for the desired behavior.
Android Studio doesn't seem to do this however. They have a default of minifyEnabled = false for projects unless flipped. Thus I suspect that Proguard is still running on their end between the two tools(javac -> dx.jar or inside jack) as the legacy Android Build Process is more or less the following:
Source Code / Library Code -> javac -> Java bytecode (.class) -> proguard -> minimized bytecode (.class) -> dex -> DEX bytecode (.dex)
Thus a form of optimization via shrinking is happening on the Android Studio side and not on our side. I believe this should happen by default for library references like Google Play Services where they provide their own proguard configuration already. That ontop of the default proguard-android.txt, we should also give the option of specifying the further optimized proguard-android-optimize.txt if a customer wanted to try that inside their project for further optimization.
We allow a custom proguard config for further control as well if it's needed past one of the two configurations noted above.
So there are two conditions for Proguard to run in a Debug configuration:
#1 You have to disable the shared runtime
#2 The linker must run (Thus it cannot be set to None)
My original attempt at documenting this feature shows that only a Release configuration can run the Proguard Task because the conditions are easier to figure out:
Thus for this use-case, one would have to enable Proguard in a Debug configuration with the conditions noted above for Proguard to properly shrink all of the .class items before sending off to dx.jar. My testing with this same project gave a total count of 21k after running proguard and dx.jar. This is approximately the same that I found in Android Studio.
I believe some discussion on what we can do to improve this experience needs to happen. Currently the experience of adding too many references in the current Debug configuration is troublesome because it requires the developer to figure out what "java.exe exited with code 2" means, and the error says to turn on multidex when in reality if we ran Proguard by default, we can avoid multidex completely in these use-cases. The foreseeable problems this can cause is unknown to me as I'm looking for some insight in that area. Also if it's possible to run Proguard with the Shared Runtime and Fast Dev, that would be another avenue to explore for Debug configuration. However it seems that $(_ProguardProjectConfiguration) doesn't populate with it enabled.
Any progress being made on this? We are running into the same issues with Xamarin hitting the dex limit way before the same kind of project using Android Studio.
This is causing a lot of churn and wasted productivity for us.
In my experience the current work around to avoiding requiring multidex being enabled is to enable Proguard in your project (multidex actually does this implicitly anyway) and have that step working properly.
In my one app, I was able to reduce the dex count by just using proguard to an acceptable limit where I otherwise was hitting the dex limit error.
We still need to investigate how to handle this in the Xamarin.Android chain (do we implicitly use proguard in the build process? how does that work for proguard config errors and warnings, etc..).
I am confirming this behavior based on my comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6 and comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c8
anyone have a fix for this? it is causing tons of wasted time and messing with the ability to hit breakpoints
I've also had issues with this recently. I had an app which was running fine without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android Support packages and started to hit the Dex limit. No new NuGet packages were introduced, just updating existing ones.
If ProGuard is going to be enabled by default, I think it should be in place, that NuGets and Xamarin Components, such as the Play Services ones add their own ProGuard configuration, similar to how it is done in Android Studio and Gradle. In most cases no one should have to mess around with those configs.
(In reply to Tomasz Cielecki from comment #11)
> I've also had issues with this recently. I had an app which was running fine
> without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android
> Support packages and started to hit the Dex limit. No new NuGet packages
> were introduced, just updating existing ones.
> If ProGuard is going to be enabled by default, I think it should be in
> place, that NuGets and Xamarin Components, such as the Play Services ones
> add their own ProGuard configuration, similar to how it is done in Android
> Studio and Gradle. In most cases no one should have to mess around with
> those configs.
I definitely agree here.
There is already work done in this area to help out: https://github.com/xamarin/XamarinComponents/pull/166
@jondouglas can this be used now? also when proguard is on by default can we still use fast deployment?
Additionally v25.4.0 and up will ship with proguard configs in the nuget packages and targets files to have them added. Eventually xamarin.android will properly detect them from .aar files too.
(In reply to dhaligas from comment #13)
> @jondouglas can this be used now? also when proguard is on by default can
> we still use fast deployment?
You can use this task as long as your Xamarin.Build.Download is 0.4.6 and above as it will contain the XamarinBuildAndroidAarProguardConfigs Task. See what Jon Dick has to say regarding shipping proguard configs in the NuGet packages as I believe that's the missing element in the current NuGets. You will see a "Proguard" folder with a proguard.txt if the library includes it's own configuration. (A quick example is the Xamarin.Android.Support.Design package 25.4.0 - https://www.nuget.org/packages/Xamarin.Android.Support.Design/25.4.0-rc1)
Unfortunately Proguard in a Debug configuration still requires the Shared Runtime to be turned off and the Linker to run to work. (Explained here: https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6)
@jondick just updated to the latest support libs and now I cannot build
Error: Error building target _XamarinAndroidBuildAarProguardConfigs: Microsoft.Build.BuildEngine.InvalidProjectFileException: Cycle in target dependencies
super disappointed this isn't getting a higher priority, and now pushed out to 15.5 :(
I agree with Malcom. I cannit use fast deployment and has doubled my android build times. Very unproductive with this issue
*** Bug 53059 has been marked as a duplicate of this bug. ***
I have added an issue at https://github.com/xamarin/xamarin-android/issues/1337.