Bug 58269 - Mono 5.0.1.1 runtime performs significantly worse than 3.8 on ARM
Summary: Mono 5.0.1.1 runtime performs significantly worse than 3.8 on ARM
Status: RESOLVED NORESPONSE
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: 5.0
Hardware: Other Linux
: --- enhancement
Target Milestone: Future Cycle (TBD)
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2017-07-20 14:07 UTC by Dirk Groot
Modified: 2017-10-11 17:51 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Dirk Groot 2017-07-20 14:07:28 UTC
### Some background

We're maintaining quite a large application for ARM hardware using Mono. Currently we use Mono 3.8. Because we want to support TLS 1.2 and because we want to stay up-to-date in general, we're trying to migrate to Mono 5.0.1.1.

### The issue

With Mono 3.8, our application performs acceptably. With Mono 5.0.1.1, the performance has degraded significantly. Symptoms are:

* Startup time has almost doubled: from about 1m27s to 2m43s
* "Random" regression test failures caused by timeouts

Because of this, Mono 5.0.1.1 is currently not usable for our application.

### Hardware/software environment

CPU as reported by dmesg:

   CPU: ARMv6-compatible processor [4117b363] revision 3 (ARMv6TEJ), cr=00c5387f
   CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache

Linux kernel    : v2.6.38.8
Physical memory : 128Mb (no swap)

Mono cross-compiled with a toolchain based on GCC version 4.9.3

For compiling Mono 3.8 we use no optimization switches.
For compiling Mono 5.0.1.1 we use --enable-small-config, --enable-minimal=aot,profiler,debug,logging,com,ssa and CFLAGS=-Os
Comment 1 Ludovic Henry 2017-07-20 14:27:42 UTC
Hello, could you please provide sample cases of what used to work on mono 3.8, and doesn't work anymore on mono 5.0? Without that it would be very hard for us to understand what's going wrong in your case. Thank you!
Comment 2 Dirk Groot 2017-07-25 13:44:17 UTC
I've done some further investigation, and it turns out that it's not as bad as I initially thought. The performance degradation is only noticable in the startup and initialization sequence of our application. There are 2 reasons for this:

First of all, with Mono 3.8 we use the MONO_XMLSERIALIZER_THS=no option, to disable the generation of XML serializers. With Mono 5, this doesn't work anymore because it uses the reference source for XmlSerializer. We're initializing quite a few XML serializers at startup, so code generation adds noticably to the startup time.

Secondly, we use SimpleInjector for dependency injection. After initializing SimpleInjector we call Container.Verify (see https://simpleinjector.org/ReferenceLibrary/html/M_SimpleInjector_Container_Verify.htm). This process takes a lot longer on Mono 5 than on Mono 3.8 (about twice as long). I haven't been able to do a thorough investigation on this yet.

As a bonus: Initial JITting of IL code also seems to be a bit slower on Mono 5 than on Mono 3.8. To demonstrate this, I've performed an application startup with 'mono --stats' on Mono 3.8 and 5. The results can be seen in the following 2 gists (look at the "Total time spent JITting" stat):

* Mono 3.8: https://gist.github.com/Dricus/3c75ac1246f3a61554c27d5d395134b6
* Mono 5: https://gist.github.com/Dricus/bf44effada4a3f657d708cc9e104d112

Maybe this difference can be (partly) explained by the fact that Mono 5 had to JIT about 2700 methods more than Mono 3.8.
Comment 3 Ludovic Henry 2017-07-25 15:16:17 UTC
Hello,

Looking at the `--stats` output you provided, the amount of JITted code is in fact much bigger in 5.0 than in 3.8. For example the `Compiled CIL code size` is 1.3M in 3.8 vs 2M in 5.0, and the `Total time spent JITting (sec)` is 77s in 3.8 and 208s in 5.0. This is most likely due to the import of sources from ReferenceSource as you pointed out.

If you can provide a sample test case exposing the initialisation's performance problems you are facing, we will take a look at it, and try to improve performance on them.

Thank you
Comment 4 Rodrigo Kumpera 2017-10-11 17:51:38 UTC
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and reopen the bug report.

Thank you!

Note You need to log in before you can comment on or make changes to this bug.