Bug 37555 - high system load -> performance degradation since 4.0
Summary: high system load -> performance degradation since 4.0
Alias: None
Product: Runtime
Classification: Mono
Component: GC ()
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Ludovic Henry
Depends on:
Reported: 2016-01-10 15:09 UTC by pr0vieh
Modified: 2016-10-18 22:23 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

cpu usage (39.07 KB, image/png)
2016-01-10 15:09 UTC, pr0vieh

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description pr0vieh 2016-01-10 15:09:20 UTC
Created attachment 14515 [details]
cpu usage

i have run a git bisect session to find the problem

362df4cc7b0d0c8b530994b819fd23a1e00c1edf is the first bad commit
commit 362df4cc7b0d0c8b530994b819fd23a1e00c1edf
Author: Zoltan Varga <vargaz@gmail.com>
Date:   Fri Aug 21 17:46:55 2015 -0400

    [runtime] Implement support for dynamic methods in stack traces in a different way: instead of constructing the stack trace eagerly when it contains a dynamic method, save the list of dynamic methods into the Exception object so they are kept alive.

the system load is 3 times higher then before, you can see this at the picture in attachment

is it possible to deactivate this feature to get back to the old performance of mono ?

e.g. with the --enable-minimal= option ?
Comment 1 Zoltan Varga 2016-01-10 23:18:53 UTC
This shouldn't happen. Could you create some kind of reproducible test case ?
Comment 2 pr0vieh 2016-01-11 01:22:37 UTC
that's difficult because I'm not a programmer, only a user
I know many web requests are executed in many Threads

i use MONO_THREADS_PER_CPU=100 for example
Comment 3 pr0vieh 2016-01-11 22:09:01 UTC
it is possible simply to turn off this feature?
then I can test whether it is really the reason

if yes how ?
a small patch would help, I think
Comment 4 Zoltan Varga 2016-01-11 22:16:17 UTC
You can try reverting the commit using
git revert -n 362df4cc7b0d0c8b530994b819fd23a1e00c1edf 
and rebuilding mono.
Comment 5 pr0vieh 2016-01-12 21:49:59 UTC
mono not building with a revert of your commit

but I have some checkouts later the problem that I get an extreme load because obviously something is wrong with the threads...

I believe this is not the commit that cause this problem
but I think it might have something to do with the MS threadpool...

then it's probably as it is

i have no idea to reproduce this... e.g. a thread stress test or so not sure

i think we close this bug for now

Thanks for the attention
Comment 7 Zoltan Varga 2016-01-22 16:58:07 UTC

The referenced commit only affects exception handling, and a well working server app shouldn't throw too many exceptions.
Comment 8 pr0vieh 2016-01-22 21:07:52 UTC
this is not a server app, this app do a mass of web requests

but you're right this commit is not the problem,for sure
I think it's the thread pool implementation
but the fact is that the performence is Decreased
it would help if I debug something, if so what and how?
Comment 9 Ludovic Henry 2016-01-25 15:48:37 UTC
From the stackoverflow information, the issue does not seem to be with the ThreadPool, but with System.Threading.Timer which seems to spend way more time at https://github.com/mono/mono/blob/master/mcs/class/corlib/System.Threading/Timer.cs#L395

As for your use of MONO_THREADS_PER_CPU=100, that means that if you have a 8 core machine (as it seems to be according to the graph you provided), the ThreadPool will have at it's disposition a minimum number of 800 threads. Even for a decent modern system, that looks like a lot.

Otherwise, which program are you running?
Comment 10 pr0vieh 2016-01-25 21:41:33 UTC
thats not all... 
this is an extreme usage of mono
i have 16 "Nodes" with Threads per CPU=100 we talk about ~ 1500 Threads in Mono 4.0 and ~ 2400 in Mono Master...

this is the Software

but you can't get "Work" for this Program without an Account and Account Creating is Deactivated

I can run some debugging stuff if you tell me what and how
Comment 11 pr0vieh 2016-10-18 22:23:48 UTC
The performance is almost back again
I'll close it
Thank you for all the work