Bug 52188 - 100% CPU Usage on Server
Summary: 100% CPU Usage on Server
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: 4.6.0 (C8)
Hardware: Other Linux
: --- normal
Target Milestone: ---
Assignee: Bugzilla
URL: https://www.spiritualselftransformati...
Depends on:
Blocks:
 
Reported: 2017-02-01 16:35 UTC by Etienne Charland
Modified: 2017-03-28 15:44 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Etienne Charland 2017-02-01 16:35:04 UTC
I'm running ASP.NET websites on a Ubuntu server hosted on Linode. I have updated Ubuntu to 16.04.1 LTS and MONO to v4.6.2

Since these updates, the websites still work, but after some time (maybe a day or two), the CPU goes to 100% usage. Looking into the process manager, I see one of the MONO websites is running in loop with high CPU usage. In the past few days, it happened twice, and on two different websites. It's totally random.

When that happens, Linode sends me an alert, I type 'htop', select the process, and kill it, and then everything is back to normal. For a day or two.

These are very basic websites, mostly plain HTML. No complex code that could run in loops, and they've been running fine for years.

What could be causing this, and how can I solve this?
Comment 1 Etienne Charland 2017-02-05 14:43:50 UTC
No comment on this?

After 2 days, EVERY MONO process is taking up the full CPU.

Looking into the Apache2 log file, I could find this related to MONO

WARNING: WebConfigurationManager's LRUcache evictions count reached its max size
Cache Size: 100 (overridable via MONO_ASPNET_WEBCONFIG_CACHESIZE)

Also, "service apache2 restart" does not solve the problem. I must manually kill the processes, or reboot.


Someone reported a similar issue here but never got a response
http://stackoverflow.com/questions/30691619/mono-mod-mono-consuming-100-of-cpu
Comment 2 Joakim 2017-02-06 08:36:01 UTC
I can trigger the bug every time with this steps:
1 - access the site (do something that activates on mono)
2 - run ‘service apache2 reload’
For each time a do these steps, I get a new “Main”-mono process at 100% CPU.
Because of Apaches daily graceful reload, I have to manually kill that process every morning.
Comment 3 Etienne Charland 2017-02-06 16:03:17 UTC
You're right... 'service apache2 reload' randomly causes MONO processes to run at 100%. First time it caused website A to go 100%. Second time was OK. Third time caused website B and C to go 100%. Furthermore, one of these websites is a test website that is never being called, and that hasn't even been JIT compiled.
Comment 4 Etienne Charland 2017-02-07 16:46:37 UTC
I uninstalled everything MONO-related and re-installed it to make sure there was no conflict issue between versions and packages. I actually wanted to try installing an older version but ran into problems, so just got the same latest version back up and running.

The problem persists. Additionally, sometimes it's websites that aren't even online that are taking 100% CPU after "service apache2 reload".

If you want to go one step further, you could play around and install older versions of MONO and see which versions are affected, since we can consistently reproduce the problem. Otherwise, I might do so later... played enough with my server this morning.

Here are the details of the problems I was having while reinstalling.
http://stackoverflow.com/questions/42094567/ubuntu-16-04-1-mod-mono-wont-install

Main thing is... mono-complete didn't install libapache2-mod-mono and it had to be installed separately.
Comment 5 Saad 2017-02-21 12:02:00 UTC
I have exactly the same problem, processes taking up 100% CPU and "evictions count reached max size" errors.

I'm using mono 4.6.0 with apache2 and mod-mono on a Debian 8.6
Comment 6 Etienne Charland 2017-03-06 15:47:10 UTC
I haven't yet tried older versions, but MONO v4.8 still has the issue. However, it seems to happen less regularly. I can now go 2-3 days without killing processes.
Comment 7 Etienne Charland 2017-03-23 20:33:28 UTC
I have played around with various MONO versions, and typing "service apache2 reload" to reproduce the high CPU usage problem.

In MONO 4.8, it seems to happen to happen a bit less often but the problem is still there.

In MONO 4.2.3.4, the problem is also there.

In MONO 4.2.1 that comes by default on Ubuntu, this problem doesn't happen.

As for .NET Core, some have tried it and highly recommended me to avoid it until it becomes more stable.

So for now, the only solution is to stick to MONO 4.2.1

This also confirms that this is related to MONO and not to my code or the server configuration.
Comment 8 Etienne Charland 2017-03-28 15:44:52 UTC
On a brand new Debian VPS, I still get the high CPU usage on 6 MONO processes this morning... Re-installing everything isn't the solution. Only solution then would be to find a version of MONO that doesn't screw this up.

Note You need to log in before you can comment on or make changes to this bug.