Bug 25683 - Per-apk library loading on bundled system Apps using Xamarin
Summary: Per-apk library loading on bundled system Apps using Xamarin
Alias: None
Product: Android
Classification: Xamarin
Component: Mono runtime / AOT Compiler ()
Version: 5.1
Hardware: Macintosh Mac OS
: Normal enhancement
Target Milestone: 5.1
Assignee: Jonathan Pryor
Depends on:
Reported: 2015-01-02 10:35 UTC by Alexandre Rocha Lima e Marcondes
Modified: 2015-02-11 11:06 UTC (History)
5 users (show)

Is this bug a regression?: ---
Last known good build:

Sample Java project with .so loading libraries correctly on Android 5.0 embedded apks (791.98 KB, application/octet-stream)
2015-01-02 10:35 UTC, Alexandre Rocha Lima e Marcondes

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Alexandre Rocha Lima e Marcondes 2015-01-02 10:35:28 UTC
Created attachment 9226 [details]
Sample Java project with .so loading libraries correctly on Android 5.0 embedded apks

When you embed a Xamarin built APK on AOSP, the monodroid runtime is not loaded properly. Up to mono 3.6 it all work from /system/lib so far. When compiled with mono 3.6 sgen is the default garbage collector and it seems there is trouble to load it from /system/lib. 

I/ActivityManager(  780): Start proc com.sikur.client.android for activity com.sikur.client.android/sikur.SplashActivity: pid=5896 uid=10041 gids={50041, 3003, 1028, 1015}
I/dalvikvm( 5896): Enabling JNI app bug workarounds for target SDK version 12...
D/dalvikvm(  177): GC_EXPLICIT freed 41K, 1% free 16692K/16768K, paused 1ms+1ms, total 16ms
D/dalvikvm(  177): GC_EXPLICIT freed <1K, 1% free 16692K/16768K, paused 1ms+1ms, total 15ms
D/dalvikvm(  177): GC_EXPLICIT freed <1K, 1% free 16692K/16768K, paused 1ms+1ms, total 14ms
F/MonoDroid( 5896): cannot find libmonosgen-2.0.so in override_dir: (null), app_libdir: /data/data/com.sikur.client.android/lib nor in dataDir as: /data/data/com.sikur.client.android/lib/libmonosgen-2.0.so
I/ActivityManager(  780): Process com.sikur.client.android (pid 5896) has died.
D/Zygote  (  177): Process 5896 exited cleanly (15)

 I have found a (commit)[https://android-review.googlesource.com/#/c/93071/] on Android AOSP repository that implements the pattern /system/lib/apk-name/ which is somewhat similar to what is used on /data/app-lib/apk-name/ , unfortunately it is on Android L Preview branch and as far as I tested it is not working on 32-bit Android distributions. So I did a Java project that has a method that could replace the default loadlibrary on Xamarin and monodroid. It detects if the apk is running as system and, if so, tried to load progressively the libraries on these directories, in order:

1. /system/lib or /system/lib64
2. /system/lib/arm or /system/lib64/arm
3. /system/lib/apk-name or /system/lib64/apk-name
4. /system/lib/apk-name/arm or /system/lib64/apk-name/arm
5. /system/lib/apk-name/abi-name or /system/lib64/apk-name/abi-name for each abi name published on Build.SUPPORTED_ABIS

I tried to reach Google regarding these commits and support for per-apk library directories on Android 5 but had no replies from them. I reached Jonathan Pryor last year by twitter and email and was waiting for the final Android 5.0 release in order to make sure how things would be handled by Google on this.

We are working with embedding Xamarin Apps on AOSP systems and this is a really important issue for us because since mono 3.6 we are not able to embed Xamarin Apps anymore.

I am available to help in any way possible to make it happen.

Happy new year guys.
Comment 1 Jonathan Pryor 2015-01-05 15:22:07 UTC
> When you embed a Xamarin built APK on AOSP,

Please please please, is there a guide/documentation/*something* on how you actually set this up? We dearly need a proper repo to actually support this scenario, and we don't have one.
Comment 2 Atsushi Eno 2015-01-06 02:34:15 UTC
You have an interesting project :-)

I find it weird that mono before 3.6 worked for you, which likely means you had mere embedded mono, not Xamarin.Android, because Xamarin.Android from its birth required sgen GC to achieve interoperability with Dalvik VM.

If that's the case, the entire support story by Xamarin (including runtime licensing) would be different. You might want to ask support@xamarin.com for such use cases.

(Setting as NEEDINFO as the actual use of Mono or Xamarin is not clear to us so far.)
Comment 3 Jonathan Pryor 2015-01-06 14:17:40 UTC
I believe they meant Xamarin 3.6 (as provided by Visual Studio), not Mono 3.6, which I believe would be Xamarin.Android 4.14, when libmonodroid.so was "split up" into separate libmonodroid.so and libmonosgen-2.0.so libraries.
Comment 4 Alexandre Rocha Lima e Marcondes 2015-01-09 12:57:03 UTC
Atsushi Eno, Jonathan Pryor is right the problem started when the updated Xamarin Studio forced the use of the "split up" library. It is fairly easy to test the problem and I have two possible ways to reproduce it: using a rooted Android device or compiling an AOSP image with the bundled Xamarin app. I can provide an Android 5 AOSP image for Google Devices ( https://source.android.com/source/building-devices.html#picking-and-building-the-configuration-that-matches-a-device ). On my environment I use a Nexus 5 device running an AOSP Android 5 image.

## If you want to test as root:

1. Root the device
2. Use adb as root:
    $ adb root
3. Remount system as read/write:
    $ adb mount -o remount,rw /system /system
4. Create the directory to put the APK (APK-name can be anything  but the apk file name has to be the same as the directory):
    $ adb shell mkdir /system/app/APK-name
5. Send the APK to /system/app/APK-name
    $ adb push /local/directory/APK-name.apk /system/app/APK-name
6. Create the directory to put the APK libraries:
    $ adb shell mkdir /system/lib/APK-name
7. Unzip the apk and extract the native libraries:
    $ unzip /local/directory/APK-name.apk -d /local/directory/APK-name
8. Send the all libraries from /local/directory/APK-name/lib/armeabi/ to /system/lib/APK-name
    $ adb push /local/directory/APK-name/lib/armeabi/libnative.so /system/lib/APK-name
9. Remount system as read-only:
    $ adb mount -o remount,ro /system /system

## If you want to test on an AOSP Image

Please reply with the link to the APK to embed on the image and the device that I should create the image. I will build it, embed the APK and publish here a link to the image to download.

P.S.: If you embed (put on /system) any app created with Xamarin it won't work due to libmonosgen-2.0 loading restrictions. If you build the attached project and embed it it loads the library correctly on different configurations if it detects that the app is running as system.
Comment 5 T.J. Purtell 2015-01-13 00:19:43 UTC
Our partners are reporting this is happening too with an Android 4.4.2 image.
> F/monodroid(10259): cannot find libmonosgen-2.0.so in override_dir: (null), app_libdir: /data/data/mobisocial.omlet/lib nor in dataDir as     : /data/data/mobisocial.omlet/lib/libmonosgen-2.0.so
Comment 6 Alexandre Rocha Lima e Marcondes 2015-01-13 07:30:52 UTC
T.J. Purtell,

It happened for us also on any Android version after Xamarin split up libmonodroid.so and libmonosgen-2.0.so libraries.

I have tested on Android 5 because this version was supposed to have support for per-apk library on /system/lib for embedded applications. I haven't tested on 64-bit versions yet to be sure, but on 32-bit versions of Android 5 it is not working. Regardless of Android support for this scheme Xamarin has to update the loading process of libmonosgen-2.0.so as far as I understood.
Comment 7 Atsushi Eno 2015-01-13 09:09:12 UTC
Those native libraries load as expected on Android devices that are in the market, including those with Android 5. Does this reproduce with Nexus 5 that is NOT rooted? If so we will have to fix it in somewhat higher priority.
If it happens ONLY with the rooted devices, what did your rooting actually change? Didn't it break native library loader?

Custom built Android, or in other words something that are not certified to be compatible with "Android" as in CTS [*1], would not be our target at this moment. For such "Android-like" target, we'd need some significant investigation that is not just like a few days bugfixes (it is not even a bug). I believe this is fair expectation as our target platform is "Android".

[*1] http://source.android.com/compatibility/cts-intro.html
Comment 8 T.J. Purtell 2015-01-13 11:17:13 UTC
Yes everything works as expected  when downloading from the Google Play store.  The scenario I mention is with preparing a firmware image that is CTS compliant, with our Xamarin Android app preinstalled into the system image.  This has the same error as Alexandre is encountering.

The steps for rooted devices is a way to manually insert the files to closely reproduce how an android firmware build process would have laid of the files from the apk.   The issue is not related to rooting.
Comment 9 Alexandre Rocha Lima e Marcondes 2015-01-13 15:17:55 UTC
Atsushi Eno,

Yes, I agree with you they do load as expected on Android devices, including
non-CTS Android builds as long as you run the APKs on the userdata partition.
As a user I disagree with you when you state that Xamarin only supports  CTS
compatible Android builds because I see a lot of talks and official messages
about Amazon Fire OS and it is not a CTS compatible Android, but ok, I won't
argue this.

Regarding the problem (I agree it is not a regular bug) it occurs on any
Android, CTS compatible or not, on any version, rooted or not. The problem only
happens when the APK runs as System, from /system/app, and have to load
libmonodroid.so and libmonosgen-2.0.so from /system/lib. As far as I understood
there are two problems on this scenario:

 1. The first library loads successfully using the regular Android native
library loader, libmonosgen-2.0.so is loaded by libmonodroid.so using dlopen on
the location provided by MonoPackageManager.LoadApplication()
(`ApplicationInfo.dataDir + "/lib"`). It doesn't exist there, so the dlopen
 2. libmonodroid.so is a moving target, so it's interface changes on each
Xamarin update. There are possible issues of having both libraries directly on
/system/lib as it might conflict with user installed Xamarin Apps on the same
Android build and the conflict would block these user installed APKs from run.

On previous talks with Jonathan Pryor he stated that this loading problem could
be easily solved if I had a way to ensure that the embedded applications could
use some structure like: /system/app/ApkName and /system/lib/ApkName.

I managed to do it on AOSP and my build now can use this scheme but
libmonodroid.so is not ready for it yet. The  attached java project
demonstrates how I did to load it and to ensure that it behaves differently
when deployed as System App and as User app. Sincerely I prefer the following
schema: /system/lib/ApkName/libmonodroid.so for now on 64-bit environments
Android 5 has /system/lib and /system/lib64 .

Well, the whole issue is although we know Xamarin focuses on APKs running on
regular Android environments, this problem turns out to be a showstopper for
programmers that are willing to provide applications for embedding on CTS
certified and non-certified Android builds. There is a relatively big market
into developing applications for embedding by OEM (Motorola, LG, etc.) and
Mobile Operators (T-Mobile, AT&T, etc.), as well as for AOSP custom builds.

Please, if Xamarin is not willing to support this by modifying the loading
process of libmonosgen-2.0.so at least open the process and particularities of
this loading process so that it may be solved on the Android side of the issue.
I, and I think most of Xamarin clients that embed APKs, prefer Xamarin dealing
with it as it may be a single patch that creates a new loading process taking
in account the System embedded applications. This new loading process may be
used everywhere on Xamarin as a Xamarin.Android native loading function for
System Apps and internally on mono runtime libraries and any other library that
may be needed on the future.
Comment 10 Atsushi Eno 2015-01-26 10:48:40 UTC
OK we made a change to also try to load libmonosgen02.0.so from /system/lib so that your expected scenario would work.

Just like Microsoft doesn't "support" ASP.NET vNext on Mac and Linux, non-"Android" should be still out of our support but we are likely continue to help.

[monodroid 9e90963]
Comment 11 Alexandre Rocha Lima e Marcondes 2015-01-26 10:56:31 UTC
I understand Atsushi, and respect it as business and strategic decisions we have to make in order to build and support a product (Xamarin on this case). Thanks for your effort.

If and when we can test it I might promptly report the outcome on my systems, will wait for a link or an update on Alpha. I just think that despite not supporting non-Android systems, the context of embedded Xamarin applications on CTS compatible Android should be supported. I sincerely think that supporting the CTS case would solve the non-CTS cases also.
Comment 12 Jonathan Pryor 2015-01-27 13:15:46 UTC
Addendum to Comment #10:

As per our current testing, this is what we've found so far. Xamarin.Android 5.0 (now in preview![0, 1]) changes native library detection to use ApplicationInfo.nativeLibraryDir [2] as the preferred directory to use for loading libmonosgen-2.0.so, and on recent Android builds (post 5.0?), this appears to actually be useful!

If you enable all logging [3] (not recommended due to volume, but bear with me):

    $ adb shell setprop debug.mono.log all

the process bootstrap code will print out all locations that it checks for libmonosgen-2.0.so until it is found:

> Trying to load sgen from: /data/data/XamarinInAospSample.XamarinInAospSample/files/.__override__/libmonosgen-2.0.so
> Trying to load sgen from: /system/app/XamarinInAospSample/lib/arm/libmonosgen-2.0.so
> Trying to load sgen from: /data/data/XamarinInAospSample.XamarinInAospSample/libmonosgen-2.0.so
> Trying to load sgen from: /system/lib/libmonosgen-2.0.so

The first entry is the fast deployment directory for Debug builds; it isn't used in Release builds.

The second entry is the ApplicationInfo.nativeLibraryDir entry.

Meaning that the current 5.0 preview *should* support being used with System apps with Android 5.0 if the appropriate directory exists, which appears to be in the pattern:

> /system/app/@SHORT_PACKAGE@/lib/@ABI@

What's odd is that @SHORT_PACKAGE@ is not the full package name -- note that it's only checking up to the first '.', not the *full* package name. This is probably an Android bug. (Someone should file that...)

Also odd is that the ABI isn't the same value used within .apk files, which would be "armeabi" or "armeabi-v7a". I'm not sure what the full set of values is.

If that directory exists, *and* it contains libmonosgen-2.0.so, then the appropriate libmonosgen-2.0.so will be loaded and used, thus avoiding "pollution" of the /system/lib directory.

[0]: http://developer.xamarin.com/releases/android/xamarin.android_5/xamarin.android_5.0/
[1]: http://developer.xamarin.com/guides/android/wear/installation/
[2]: http://developer.android.com/reference/android/content/pm/ApplicationInfo.html#nativeLibraryDir
[3]: All logging may be disabled by clearing the system property:

    adb shell setprop debug.mono.log ''
Comment 13 T.J. Purtell 2015-01-27 13:41:11 UTC
Thanks for he fix and the detailed info.
Comment 14 Alexandre Rocha Lima e Marcondes 2015-01-27 14:52:21 UTC
Guys & Girls from Xamarin thank you very much for your patience and effort.

I confirm that the second option ( /system with per-APK lib)  loaded perfectly on my Android build. I would like to give information on some aspects though, but just for the knowledge, the patch is perfect for me and my company.

When you state that the pattern is: /system/app/@SHORT_PACKAGE@/lib/@ABI@ , according to the Android Package Manager source code I had researched:

* @SHORT_PACKAGE@ seems to be the last part of the full package name. On your example the first and last parts have the same name. I think that if you get a package with different parts it will be only the last part. Additionally, AOSP practice on the Google maintained packages is a short name using camel case without dots and the APK inside it has the same name as the directory.

* @ABI@ seems to be only: arm, x86 or mips when used on the system library context. Of course now there are 64-bit versions of the first two. I understand that Xamarin, on the version we are working on, don't support 64-bit Android applications yet as I didn't see any reference for it on the IDE or 64-bit versions of the runtime being included on the APKs. According to the source codes the full set of options for system should not be differentiated between 32 and 64-bit versions on the ABI level, but on the lib directory name: /system/app/@SHORT_PACKAGE@/lib/@ABI@ or /system/app/@SHORT_PACKAGE@/lib64/@ABI@

P.S.: If possible, support for APKs installed on /oem and /vendor for Android 5.0 CTS builds would be great for the future. I mean, as you are working on this loading code it should be a small addition with huge impact in the future. I guess that Google introduced it in order to differentiate system apps and third-party apps (vendor or oem).

[0] Support per-package lib dirs for bundled apps: https://android.googlesource.com/platform/frameworks/base/+/addfbdc%5E!/
[1] Correction for the removal of APK libraries: https://android.googlesource.com/platform/frameworks/base/+/2020084%5E!/
[2] /system/lib/* path handling bug: https://android.googlesource.com/platform/frameworks/base/+/c38a807%5E!/
[3] Multi-arch support: https://android.googlesource.com/platform/frameworks/base/+/7dba6eb%5E!/
[4] /oem and /vendor support: https://android.googlesource.com/platform/frameworks/base/+/740888f%5E!/
[5] Multi-arch application install: https://android.googlesource.com/platform/frameworks/base/+/ff110bd%5E!/
Comment 15 Jonathan Pryor 2015-01-27 14:58:56 UTC
@Alexandre: I'm hoping that *no* changes will be required on our part for vendor/oem support. (I also suspect my hopes will be dashed.)

As per Comment #11, Xamarin.Android 5.0 uses ApplicationInfo.nativeLibraryDir to find native libraries. So long as Android provides the correct directory within ApplicationInfo.nativeLibraryDir, it shouldn't matter where that directory is.

Thus, presumably/hopefully, if an app is installed into /vendor, Android will provide a `/vendor/app/@SHORT_NAME@/lib/@ABI@` path for ApplicationInfo.nativeLibraryDir, and (so long as libmonosgen-2.0.so is installed in the "right" magic directory), everything will Just Work.

What I don't know is whether Android does the right thing regarding ApplicationInfo.nativeLibraryDir and vendor/oem package installs.