### Some background
We're maintaining quite a large application for ARM hardware using Mono. Currently we use Mono 3.8. Because we want to support TLS 1.2 and because we want to stay up-to-date in general, we're trying to migrate to Mono 22.214.171.124.
### The issue
With Mono 3.8, our application performs acceptably. With Mono 126.96.36.199, the performance has degraded significantly. Symptoms are:
* Startup time has almost doubled: from about 1m27s to 2m43s
* "Random" regression test failures caused by timeouts
Because of this, Mono 188.8.131.52 is currently not usable for our application.
### Hardware/software environment
CPU as reported by dmesg:
CPU: ARMv6-compatible processor [4117b363] revision 3 (ARMv6TEJ), cr=00c5387f
CPU: VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Linux kernel : v184.108.40.206
Physical memory : 128Mb (no swap)
Mono cross-compiled with a toolchain based on GCC version 4.9.3
For compiling Mono 3.8 we use no optimization switches.
For compiling Mono 220.127.116.11 we use --enable-small-config, --enable-minimal=aot,profiler,debug,logging,com,ssa and CFLAGS=-Os
Hello, could you please provide sample cases of what used to work on mono 3.8, and doesn't work anymore on mono 5.0? Without that it would be very hard for us to understand what's going wrong in your case. Thank you!
I've done some further investigation, and it turns out that it's not as bad as I initially thought. The performance degradation is only noticable in the startup and initialization sequence of our application. There are 2 reasons for this:
First of all, with Mono 3.8 we use the MONO_XMLSERIALIZER_THS=no option, to disable the generation of XML serializers. With Mono 5, this doesn't work anymore because it uses the reference source for XmlSerializer. We're initializing quite a few XML serializers at startup, so code generation adds noticably to the startup time.
Secondly, we use SimpleInjector for dependency injection. After initializing SimpleInjector we call Container.Verify (see https://simpleinjector.org/ReferenceLibrary/html/M_SimpleInjector_Container_Verify.htm). This process takes a lot longer on Mono 5 than on Mono 3.8 (about twice as long). I haven't been able to do a thorough investigation on this yet.
As a bonus: Initial JITting of IL code also seems to be a bit slower on Mono 5 than on Mono 3.8. To demonstrate this, I've performed an application startup with 'mono --stats' on Mono 3.8 and 5. The results can be seen in the following 2 gists (look at the "Total time spent JITting" stat):
* Mono 3.8: https://gist.github.com/Dricus/3c75ac1246f3a61554c27d5d395134b6
* Mono 5: https://gist.github.com/Dricus/bf44effada4a3f657d708cc9e104d112
Maybe this difference can be (partly) explained by the fact that Mono 5 had to JIT about 2700 methods more than Mono 3.8.
Looking at the `--stats` output you provided, the amount of JITted code is in fact much bigger in 5.0 than in 3.8. For example the `Compiled CIL code size` is 1.3M in 3.8 vs 2M in 5.0, and the `Total time spent JITting (sec)` is 77s in 3.8 and 208s in 5.0. This is most likely due to the import of sources from ReferenceSource as you pointed out.
If you can provide a sample test case exposing the initialisation's performance problems you are facing, we will take a look at it, and try to improve performance on them.
We have not received the requested information. If you are still experiencing this issue please provide all the requested information and reopen the bug report.