Bug 55117 - Xamarin.Android projects hit the multidex limit much quicker than Android Studio projects
Summary: Xamarin.Android projects hit the multidex limit much quicker than Android Stu...
Status: CONFIRMED
Alias: None
Product: Android
Classification: Xamarin
Component: General (show other bugs)
Version: 7.2 (15.1)
Hardware: PC Windows
: --- normal
Target Milestone: 15.6
Assignee: Jonathan Pryor
URL:
: 53059 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-04-12 18:50 UTC by Jon Douglas [MSFT]
Modified: 2017-10-25 11:55 UTC (History)
15 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Jon Douglas [MSFT] 2017-04-12 18:50:31 UTC
*Description:

I've been investigating a potential issue with regards to the multidex limit in Xamarin.Android projects versus Android Studio projects.

What I've noticed is that Android Studio projects seem to handle many more library references than the Xamarin.Android counterpart.

It seems that the `CompileToDalvik` MSBuild task that invokes dx.jar causes Xamarin.Android projects to reach the multidex limit faster than the Android Studio counterpart.

*Reproduction:

For my samples, I took a blank Xamarin.Android project and a blank Android Studio project.

I then added as many Android Support Libraries as I could until I hit the dex limit. Here is what I found out:

1. Xamarin.Android was only able to hold 24 references to support libraries before hitting the dex limit(Libraries have different dex counts, but using this number as reference to Android Studio):

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Xamarin.Android.Support.Animated.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Compat" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Core.UI" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Core.Utils" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.CustomTabs" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Design" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Exif" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Fragment" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Media.Compat" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Percent" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Recommendation" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Transition" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v13" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v14.Preference" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v4" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.AppCompat" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.CardView" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.GridLayout" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.MediaRouter" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.Palette" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.Preference" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.v7.RecyclerView" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Android.Support.Vector.Drawable" version="25.1.1" targetFramework="monoandroid71" />
  <package id="Xamarin.Build.Download" version="0.4.3" targetFramework="monoandroid71" />
</packages>

With these packages all added, dx.jar will fail stating that multidex needs to be enabled:

https://gist.github.com/JonDouglas/73a2562e5a658ed41885904e3385c564

Thus it claims that 67288 references are being used here from the output of dx.jar. The other strange observation here is that some of these libraries that have very different counts end up becoming the same number. You will see a repeated "2422" as a count for many libraries when the actual .jar differs.

2. Android Studio had no problem with the same packages, and it didn't even get close to the dex limit:

    compile 'com.android.support:support-compat:25.3.1'
    compile 'com.android.support:support-core-ui:25.3.1'
    compile 'com.android.support:support-core-utils:25.3.1'
    compile 'com.android.support:customtabs:25.3.1'
    compile 'com.android.support:design:25.3.1'
    compile 'com.android.support:exifinterface:25.3.1'
    compile 'com.android.support:support-fragment:25.3.1'
    compile 'com.android.support:percent:25.3.1'
    compile 'com.android.support:support-media-compat:25.3.1'
    compile 'com.android.support:support-v4:25.3.1'
    compile 'com.android.support:appcompat-v7:25.3.1'
    compile 'com.android.support:cardview-v7:25.3.1'
    compile 'com.android.support:gridlayout-v7:25.3.1'
    compile 'com.android.support:mediarouter-v7:25.3.1'
    compile 'com.android.support:palette-v7:25.3.1'
    compile 'com.android.support:recyclerview-v7:25.3.1'
    compile 'com.android.support:preference-v7:25.3.1'
    compile 'com.android.support:support-v13:25.3.1'
    compile 'com.android.support:preference-v14:25.3.1'
    compile 'com.android.support:support-annotations:25.3.1'
    compile 'com.android.support:leanback-v17:25.3.1'
    compile 'com.android.support:recommendation:25.3.1'
    compile 'com.android.support:preference-leanback-v17:25.3.1'
    compile 'com.android.support:support-media-compat:25.3.1'
    compile 'com.android.support:support-annotations:25.3.1'

`classes.dex` count: 25966

*Investigation:

However I then investigated each individual `classes.jar` being inputted in the dx.jar command from the gist above and noticed that the counts weren't even close.

https://gist.github.com/JonDouglas/e576a023cfd7f014035e86b0945b3936

This gist shows an output of 21696.

The other fact is that if you remove a couple libraries from NuGet for the Xamarin.Android application and successfully rebuild, the dex count is still around the ~22k count. This isn't even close to the 65,536 limit and thus I suspect something is going on. I haven't been able to figure out what that is quite yet. I even went as far as adding --verbose logging to dx.jar and seeing if it was adding certain libraries multiple times.

https://gist.github.com/JonDouglas/2b5eb5fc3bb2a4a390cf879be1aedbe8

However I did not see anything that quite caught my eye here. My overall suspicions:

1) Something is being added to dx.jar many times to the point of hitting the dex limit
2) Optimization/pre-dexing is different in Android Studio vs. Xamarin.Android
3) The strange number of "2422" for libraries that have different counts seems suspicious

*Version Information:

Microsoft Visual Studio Enterprise 2017 
Version 15.1 (26403.0) Release
VisualStudio.15.Release/15.1.0+26403.0
Microsoft .NET Framework
Version 4.6.01586

Installed Version: Enterprise

Xamarin   4.4.0.34 (3f99c5a)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   7.2.0.7 (b16fb82)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   10.8.0.174 (7656cc6)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.
Comment 2 Peter Major 2017-04-19 10:36:20 UTC
I'm also seeing this. It's impossible to update from AppCompat 23.3.0 to 25.1.1 or 25.3.x without hitting the limit. 

This seems like a massive problem!

The number of packages with counts exactly the same is very suspicious. My guess is that it's including the count for the dependencies (i.e. for every package using 'android.support.compat', the package's method count includes the methods in 'android.support.compat')

trouble writing output: Too many field references: 78376; max is 65536.
You may try using --multi-dex option.
References by package:
     3 android.accounts
    30 android.app
     1 android.content
    37 android.content.pm
    10 android.content.res
     3 android.database
    46 android.graphics
     1 android.graphics.drawable
     2 android.media
     1 android.media.browse
     2 android.net
    25 android.os
     3 android.print
     1 android.provider
     8 android.runtime
     1 android.support.annotation
  2656 android.support.compat
  2656 android.support.coreui
  2656 android.support.coreutils
  2760 android.support.customtabs
  2656 android.support.design
   107 android.support.design.internal
   700 android.support.design.widget
  2656 android.support.fragment
  2815 android.support.graphics.drawable
  2656 android.support.graphics.drawable.animated
  2656 android.support.mediacompat
  2835 android.support.transition
  2656 android.support.v4
    13 android.support.v4.accessibilityservice
    17 android.support.v4.animation
   885 android.support.v4.app
   114 android.support.v4.content
     1 android.support.v4.content.pm
     1 android.support.v4.content.res
    11 android.support.v4.graphics
    41 android.support.v4.graphics.drawable
     4 android.support.v4.hardware.display
    12 android.support.v4.hardware.fingerprint
    10 android.support.v4.internal.view
   388 android.support.v4.media
   379 android.support.v4.media.session
    10 android.support.v4.net
    19 android.support.v4.os
    63 android.support.v4.print
    10 android.support.v4.provider
    55 android.support.v4.text
     6 android.support.v4.text.util
   118 android.support.v4.util
   434 android.support.v4.view
   138 android.support.v4.view.accessibility
     8 android.support.v4.view.animation
   495 android.support.v4.widget
   605 android.support.v7.app
  2656 android.support.v7.appcompat
  2656 android.support.v7.cardview
     8 android.support.v7.content.res
    88 android.support.v7.graphics
    18 android.support.v7.graphics.drawable
   437 android.support.v7.media
  2656 android.support.v7.mediarouter
  2656 android.support.v7.palette
  2656 android.support.v7.recyclerview
     1 android.support.v7.text
     2 android.support.v7.transition
   124 android.support.v7.util
    77 android.support.v7.view
   249 android.support.v7.view.menu
  1483 android.support.v7.widget
    89 android.support.v7.widget.helper
     1 android.support.v7.widget.util
     6 android.text
     1 android.text.util
     2 android.transition
    14 android.util
    19 android.view
     8 android.view.accessibility
     1 android.view.inputmethod
    15 android.widget
  2650 com.google.android.gms
    20 com.google.android.gms.actions
    13 com.google.android.gms.ads.identifier
    83 com.google.android.gms.appdatasearch
  2676 com.google.android.gms.appindexing
    53 com.google.android.gms.auth
     7 com.google.android.gms.auth.api.credentials
    57 com.google.android.gms.auth.api.signin
    13 com.google.android.gms.auth.api.signin.internal
    75 com.google.android.gms.auth.firstparty.shared
  2650 com.google.android.gms.base
    38 com.google.android.gms.clearcut
    94 com.google.android.gms.common
    87 com.google.android.gms.common.api
   179 com.google.android.gms.common.api.internal
    42 com.google.android.gms.common.data
    49 com.google.android.gms.common.images
   171 com.google.android.gms.common.internal
     1 com.google.android.gms.common.internal.safeparcel
     4 com.google.android.gms.common.server
    12 com.google.android.gms.common.server.converter
    33 com.google.android.gms.common.server.response
    62 com.google.android.gms.common.stats
    27 com.google.android.gms.dynamic
     2 com.google.android.gms.dynamite.descriptors.com.google.android.gms.flags
    16 com.google.android.gms.flags.impl
  2753 com.google.android.gms.gcm
    47 com.google.android.gms.iid
   372 com.google.android.gms.internal
  2666 com.google.android.gms.measurement
   350 com.google.android.gms.measurement.internal
    32 com.google.android.gms.playlog.internal
    14 com.google.android.gms.search
     6 com.google.android.gms.security
    26 com.google.android.gms.signin.internal
  2650 com.pelicanexchange
  2799 com.roughike.bottombar
  2893 com.theartofdev.edmodo.cropper
     1 com.xamarin.forms.platform.android
     2 default
     4 java.lang
     7 java.lang.annotation
     1 java.nio
     6 java.util
     3 java.util.concurrent
     1 javax.microedition.khronos.egl
     2 md507ae2fe0c5e6c7f1cce2073326250bfa
     2 md50bd63072d32fd72e10115bcaf84d340f
    36 md5270abb39e60627f0f200893b490a1ade
     8 md531f8cc8d4d39b2fa02686ee0eb368905
     4 md534e018afb580d217ff06dd0dbae3d572
     4 md5539939fcd5322368b18e9187b5e77d3d
    14 md56987f6cc1cad2bad7d1513f02edd2700
    54 md5716b2bf943f46e4067e009323bbbb898
     6 md584f84b531215f3b805721fa27da129c9
     4 md59f70a99687498e7ba187118950981d26
    12 md5a104545e4d19c4ffe9ec3d5074a3b979
     6 md5a4ed55898dc7fb7b87f5830ba08179d9
     2 md5afd03715c12d7cea439a2ca4d0e82348
   160 md5b60ffeb829f638581ab2bb9b1a7f4f3f
    34 md5c25e555921ab1c976dcce693fa7017e6
     2 md5cf6f91d9bcbb245b12b8f240e1ec031d
    30 md5f9b7691a3024e3da489938deb6fbfb67
     6 mono
     1 mono.android
     4 mono.android.accessibilityservice
     2 mono.android.accounts
    12 mono.android.animation
    31 mono.android.app
     2 mono.android.bluetooth
    20 mono.android.content
     2 mono.android.database.sqlite
     6 mono.android.drm
     6 mono.android.gesture
     2 mono.android.graphics
     2 mono.android.graphics.drawable
     8 mono.android.hardware
     2 mono.android.hardware.display
     2 mono.android.hardware.input
     2 mono.android.inputmethodservice
     8 mono.android.location
    60 mono.android.media
    16 mono.android.media.audiofx
     2 mono.android.media.effect
     2 mono.android.media.midi
     2 mono.android.media.session
     2 mono.android.media.tv
     2 mono.android.net
     6 mono.android.net.nsd
     2 mono.android.net.sip
    18 mono.android.net.wifi.p2p
     2 mono.android.nfc
    10 mono.android.os
    10 mono.android.preference
     2 mono.android.renderscript
     6 mono.android.runtime
     6 mono.android.sax
     2 mono.android.speech
     4 mono.android.speech.tts
    12 mono.android.support.design.widget
     2 mono.android.support.transition
     4 mono.android.support.v4.app
     4 mono.android.support.v4.content
     2 mono.android.support.v4.media.session
     2 mono.android.support.v4.os
    18 mono.android.support.v4.view
     4 mono.android.support.v4.view.accessibility
    12 mono.android.support.v4.widget
     6 mono.android.support.v7.app
     2 mono.android.support.v7.graphics
     2 mono.android.support.v7.media
    32 mono.android.support.v7.widget
     2 mono.android.text
     2 mono.android.transition
    72 mono.android.view
     4 mono.android.view.accessibility
     2 mono.android.view.animation
     2 mono.android.view.textservice
     8 mono.android.webkit
    72 mono.android.widget
     2 mono.com.google.android.gms.common.api
     2 mono.com.google.android.gms.common.images
     2 mono.com.google.android.gms.security
     4 mono.com.roughike.bottombar
     8 mono.com.theartofdev.edmodo.cropper
     4 mono.java.lang
     2 mono.java.util
     2 mono.javax.xml.transform
     2 mono.net.hockeyapp.android
  2839 net.hockeyapp.android
     2 net.hockeyapp.android.adapters
    65 net.hockeyapp.android.metrics
    82 net.hockeyapp.android.metrics.model
    79 net.hockeyapp.android.objects
    80 net.hockeyapp.android.tasks
    86 net.hockeyapp.android.utils
    39 net.hockeyapp.android.views
     2 opentk
     2 opentk.platform.android
     2 opentk_1_0
     2 opentk_1_0.platform.android
     2 pelican.ui.droid.controls
     2 pelican.ui.droid.renderers
Comment 3 dhaligas 2017-05-24 19:40:47 UTC
we ran into this as well.  hoping xamarin can shed some light on this
Comment 4 dhaligas 2017-05-30 14:48:09 UTC
@jonp any updates on this?
Comment 5 Jon Dick 2017-06-01 14:21:13 UTC
I have an app that I can easily reproduce this in, what's interesting, is as soon as I run proguard, I now seem to get a dex file with a ~30,000 count, no multidex required.  If I turn proguard off, I get the error.

I don't believe this was the case on previous versions of Xamarin.Android, perhaps it's worth retesting on current stable?
Comment 6 Jon Douglas [MSFT] 2017-06-01 20:39:07 UTC
Although we provide a Proguard target after the Javac task and before the CompileToDalvik task, it is an "opt-in" situation meaning that one has to enable Proguard for the desired behavior.

Android Studio doesn't seem to do this however. They have a default of minifyEnabled = false for projects unless flipped. Thus I suspect that Proguard is still running on their end between the two tools(javac -> dx.jar or inside jack) as the legacy Android Build Process is more or less the following:

Source Code / Library Code -> javac -> Java bytecode (.class) -> proguard -> minimized bytecode (.class) -> dex -> DEX bytecode (.dex)

Thus a form of optimization via shrinking is happening on the Android Studio side and not on our side. I believe this should happen by default for library references like Google Play Services where they provide their own proguard configuration already. That ontop of the default proguard-android.txt, we should also give the option of specifying the further optimized proguard-android-optimize.txt if a customer wanted to try that inside their project for further optimization.

We allow a custom proguard config for further control as well if it's needed past one of the two configurations noted above.

So there are two conditions for Proguard to run in a Debug configuration:

#1 You have to disable the shared runtime
#2 The linker must run (Thus it cannot be set to None)

My original attempt at documenting this feature shows that only a Release configuration can run the Proguard Task because the conditions are easier to figure out:

https://developer.xamarin.com/guides/android/deployment,_testing,_and_metrics/proguard/#using

Thus for this use-case, one would have to enable Proguard in a Debug configuration with the conditions noted above for Proguard to properly shrink all of the .class items before sending off to dx.jar. My testing with this same project gave a total count of 21k after running proguard and dx.jar. This is approximately the same that I found in Android Studio.

I believe some discussion on what we can do to improve this experience needs to happen. Currently the experience of adding too many references in the current Debug configuration is troublesome because it requires the developer to figure out what "java.exe exited with code 2" means, and the error says to turn on multidex when in reality if we ran Proguard by default, we can avoid multidex completely in these use-cases. The foreseeable problems this can cause is unknown to me as I'm looking for some insight in that area. Also if it's possible to run Proguard with the Shared Runtime and Fast Dev, that would be another avenue to explore for Debug configuration. However it seems that $(_ProguardProjectConfiguration) doesn't populate with it enabled.
Comment 7 pragma.mobilexp 2017-06-29 12:48:27 UTC
Any progress being made on this?  We are running into the same issues with Xamarin hitting the dex limit way before the same kind of project using Android Studio. 
This is causing a lot of churn and wasted productivity for us.
Comment 8 Jon Dick 2017-06-29 13:12:48 UTC
In my experience the current work around to avoiding requiring multidex being enabled is to enable Proguard in your project (multidex actually does this implicitly anyway) and have that step working properly.

In my one app, I was able to reduce the dex count by just using proguard to an acceptable limit where I otherwise was hitting the dex limit error.

We still need to investigate how to handle this in the Xamarin.Android chain (do we implicitly use proguard in the build process? how does that work for proguard config errors and warnings, etc..).
Comment 9 Jon Douglas [MSFT] 2017-07-05 18:17:37 UTC
I am confirming this behavior based on my comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6 and comments in https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c8
Comment 10 dhaligas 2017-07-06 23:55:10 UTC
anyone have a fix for this?  it is causing tons of wasted time and messing with the ability to hit breakpoints
Comment 11 Tomasz Cielecki 2017-08-07 07:27:32 UTC
I've also had issues with this recently. I had an app which was running fine without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android Support packages and started to hit the Dex limit. No new NuGet packages were introduced, just updating existing ones.

If ProGuard is going to be enabled by default, I think it should be in place, that NuGets and Xamarin Components, such as the Play Services ones add their own ProGuard configuration, similar to how it is done in Android Studio and Gradle. In most cases no one should have to mess around with those configs.
Comment 12 Jon Douglas [MSFT] 2017-08-07 16:06:20 UTC
(In reply to Tomasz Cielecki from comment #11)
> I've also had issues with this recently. I had an app which was running fine
> without Multi-Dex enabled. Then I updated Xamarin and the Xamarin Android
> Support packages and started to hit the Dex limit. No new NuGet packages
> were introduced, just updating existing ones.
> 
> If ProGuard is going to be enabled by default, I think it should be in
> place, that NuGets and Xamarin Components, such as the Play Services ones
> add their own ProGuard configuration, similar to how it is done in Android
> Studio and Gradle. In most cases no one should have to mess around with
> those configs.

I definitely agree here.

There is already work done in this area to help out: https://github.com/xamarin/XamarinComponents/pull/166
Comment 13 dhaligas 2017-08-07 16:08:36 UTC
@jondouglas can this be used now?  also when proguard is on by default can we still use fast deployment?
Comment 14 Jon Dick 2017-08-07 16:15:46 UTC
Additionally v25.4.0 and up will ship with proguard configs in the nuget packages and targets files to have them added. Eventually xamarin.android will properly detect them from .aar files too.
Comment 15 Jon Douglas [MSFT] 2017-08-07 16:23:38 UTC
(In reply to dhaligas from comment #13)
> @jondouglas can this be used now?  also when proguard is on by default can
> we still use fast deployment?

You can use this task as long as your Xamarin.Build.Download is 0.4.6 and above as it will contain the XamarinBuildAndroidAarProguardConfigs Task. See what Jon Dick has to say regarding shipping proguard configs in the NuGet packages as I believe that's the missing element in the current NuGets. You will see a "Proguard" folder with a proguard.txt if the library includes it's own configuration. (A quick example is the Xamarin.Android.Support.Design package 25.4.0 - https://www.nuget.org/packages/Xamarin.Android.Support.Design/25.4.0-rc1)

Unfortunately Proguard in a Debug configuration still requires the Shared Runtime to be turned off and the Linker to run to work. (Explained here: https://bugzilla.xamarin.com/show_bug.cgi?id=55117#c6)
Comment 16 dhaligas 2017-08-09 16:35:17 UTC
@jondick just updated to the latest support libs and now I cannot build 

https://forums.xamarin.com/discussion/98777/invalidprojectfileexception-cycle-in-target-dependencies-detected-build-failed

Error: Error building target _XamarinAndroidBuildAarProguardConfigs: Microsoft.Build.BuildEngine.InvalidProjectFileException: Cycle in target dependencies
Comment 17 Malcolm Jack 2017-08-15 21:03:07 UTC
super disappointed this isn't getting a higher priority, and now pushed out to 15.5 :(
Comment 18 dhaligas 2017-08-15 21:47:44 UTC
I agree with Malcom.  I cannit use fast deployment and has doubled my android build times.  Very unproductive with this issue
Comment 19 Jon Dick 2017-09-28 14:26:54 UTC
*** Bug 53059 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.