Bug 58269 - Mono runtime performs significantly worse than 3.8 on ARM
Summary: Mono runtime performs significantly worse than 3.8 on ARM
Alias: None
Product: Runtime
Classification: Mono
Component: General ()
Version: 5.0 (2017-02)
Hardware: Other Linux
: --- enhancement
Target Milestone: Future Cycle (TBD)
Assignee: Bugzilla
Depends on:
Reported: 2017-07-20 14:07 UTC by Dirk Groot
Modified: 2017-10-11 17:51 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Dirk Groot 2017-07-20 14:07:28 UTC
### Some background

We're maintaining quite a large application for ARM hardware using Mono. Currently we use Mono 3.8. Because we want to support TLS 1.2 and because we want to stay up-to-date in general, we're trying to migrate to Mono

### The issue

With Mono 3.8, our application performs acceptably. With Mono, the performance has degraded significantly. Symptoms are:

* Startup time has almost doubled: from about 1m27s to 2m43s
* "Random" regression test failures caused by timeouts

Because of this, Mono is currently not usable for our application.

### Hardware/software environment

CPU as reported by dmesg:

   CPU: ARMv6-compatible processor [4117b363] revision 3 (ARMv6TEJ), cr=00c5387f
   CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache

Linux kernel    : v2.6.38.8
Physical memory : 128Mb (no swap)

Mono cross-compiled with a toolchain based on GCC version 4.9.3

For compiling Mono 3.8 we use no optimization switches.
For compiling Mono we use --enable-small-config, --enable-minimal=aot,profiler,debug,logging,com,ssa and CFLAGS=-Os
Comment 1 Ludovic Henry 2017-07-20 14:27:42 UTC
Hello, could you please provide sample cases of what used to work on mono 3.8, and doesn't work anymore on mono 5.0? Without that it would be very hard for us to understand what's going wrong in your case. Thank you!
Comment 2 Dirk Groot 2017-07-25 13:44:17 UTC
I've done some further investigation, and it turns out that it's not as bad as I initially thought. The performance degradation is only noticable in the startup and initialization sequence of our application. There are 2 reasons for this:

First of all, with Mono 3.8 we use the MONO_XMLSERIALIZER_THS=no option, to disable the generation of XML serializers. With Mono 5, this doesn't work anymore because it uses the reference source for XmlSerializer. We're initializing quite a few XML serializers at startup, so code generation adds noticably to the startup time.

Secondly, we use SimpleInjector for dependency injection. After initializing SimpleInjector we call Container.Verify (see https://simpleinjector.org/ReferenceLibrary/html/M_SimpleInjector_Container_Verify.htm). This process takes a lot longer on Mono 5 than on Mono 3.8 (about twice as long). I haven't been able to do a thorough investigation on this yet.

As a bonus: Initial JITting of IL code also seems to be a bit slower on Mono 5 than on Mono 3.8. To demonstrate this, I've performed an application startup with 'mono --stats' on Mono 3.8 and 5. The results can be seen in the following 2 gists (look at the "Total time spent JITting" stat):

* Mono 3.8: https://gist.github.com/Dricus/3c75ac1246f3a61554c27d5d395134b6
* Mono 5: https://gist.github.com/Dricus/bf44effada4a3f657d708cc9e104d112

Maybe this difference can be (partly) explained by the fact that Mono 5 had to JIT about 2700 methods more than Mono 3.8.
Comment 3 Ludovic Henry 2017-07-25 15:16:17 UTC

Looking at the `--stats` output you provided, the amount of JITted code is in fact much bigger in 5.0 than in 3.8. For example the `Compiled CIL code size` is 1.3M in 3.8 vs 2M in 5.0, and the `Total time spent JITting (sec)` is 77s in 3.8 and 208s in 5.0. This is most likely due to the import of sources from ReferenceSource as you pointed out.

If you can provide a sample test case exposing the initialisation's performance problems you are facing, we will take a look at it, and try to improve performance on them.

Thank you
Comment 4 Rodrigo Kumpera 2017-10-11 17:51:38 UTC
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and reopen the bug report.

Thank you!