Bug 43765 - CPU usage spikes up to 100% when accessing HTTPS endpoints on single core device
Summary: CPU usage spikes up to 100% when accessing HTTPS endpoints on single core device
Status: CONFIRMED
Alias: None
Product: Runtime
Classification: Mono
Component: General (show other bugs)
Version: 4.2.1 (C6SR0)
Hardware: Other Linux
: Normal normal
Target Milestone: Future Cycle (TBD)
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2016-08-25 22:41 UTC by Ramtin Kermani
Modified: 2017-10-11 17:38 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Ramtin Kermani 2016-08-25 22:41:19 UTC
*** Summary of the issue ***
Initially, running a simple self-hosting OWIN application that serves HTTP(S) endpoints, we observed (when using HttpS vs Http) the CPU usage on the device spikes up to 100% every 30 seconds, stays up for about 10 seconds and then falls back to about 10% which is the normal amount. The simple application serves an HTML page that makes ajax calls to the REST endpoints exposed by the application and pulls some data every 2-3 seconds. The same application does not behave this way when we switch to HTTP (instead of HTTPS) and there are no spikes. To determine if Mono implementation of HTTPS is the problem or the OWIN/WebAPI, we eliminated OWIN and created a "Simple HTTP listener" that simply listens to an Http(S) connection and serves the endpoints. We again observed the issue. Note that this problem does not occur on a 64 bit desktop Ubuntu machine with 4 cores.

*** System Characteristics ***
- An embedded device running a custom Yocto Linux
- Application runs inside an LXC container which is hosted inside a Linux VM
- Single core 1.25 GHz 32 bit Atom CPU
- 360 MB of RAM
- Mono Version: 4.2.1.102

*** Reproducing the issue ***
We have created a simple application that you can build and run on a "preferably" single core Linux device (rather than a 64 bit desktop Linux). You may emulate the device using something like Qemu. The application code is placed at Github: https://github.com/ramtinkermani/SimpleHttpListener-CSharp

*** Configuring the application ***
- You can configure the HTTP(S) port and the protocol (HTTP vs HTTPS) in the webconfig.json file. The host name is irrelevant here.
- If using HTTPS, you need to create a self-signed Server certificate for the server.
- Also Mono implementation of the HTTPS requests the Client to send a (Any!) Client certificates to the server in order to establish a secure communication. So you need to either create a Client certificate and import it into your browser or use an existing Client Certificate that is already available in your browser.

*** Profiling ***
To observe the spikes, once you run the application you can find the PID of the process using the following command: 
$ pgrep mono and then use that PID in the following command to print out the CPU/Memory usage. You can either write that data to a file for a couple of minutes or simply keep an eye on the numbers to observe the CPU spiking up to 100% every ~30 seconds. 
$ top -p <PID>
Comment 1 Alex Rønne Petersen 2016-08-29 06:03:58 UTC
Hi Ramtin,

Given that this only occurs on a very specific hardware/OS configuration that we don't have available and can't easily get access to, we really need more specific instructions on how to reproduce this under something like qemu. What exact options should we use to match as closely as possible your actual hardware? What OS and userland should we use?

Since this is a bug that only occurs on very niche hardware, it's hard for us to justify spending a ton of time trying to replicate your environment. If you can give us clear instructions on how to do so, though, it would be much easier for us to actually debug and fix the issue.

Additionally/alternatively, if you're feeling adventurous, you could try AOTing all assemblies in your application and running Mono with the `perf` tool. This could give us some idea of where all this CPU time is being spent. (I would normally suggest using the Mono profiler, but 4.2.1 unfortunately has a lot of stability bugs in the profiler that have since been fixed. You could still give it a shot, though.)
Comment 2 Ramtin Kermani 2016-09-13 23:22:03 UTC
Hi Alex,

We created a Qemu image that to a good extent emulates our device. We have included the Simple application. Here are the steps to reproduce the issue and investigate the performance:

1- Download the Qemu image from the following address:
  https://www.dropbox.com/s/xsf2scp8phcok5z/QemuVM.tar.gz?dl=0

which contains :
  - The Kernel
  - root filesystem
  - run script (A one-liner script to avoid typing the long qemu run command!)
  - Mono is already installed and in Path
  - Simple application

2- Uncompress the tarball:
  $ tar xvfz QemuVM.tr.gz

3- make sure Qemu is installed on your machine (Mine is a 32bit Ubuntu 14.04)
  $ sudo apt-get install qemu-system-x86

4- It is assumed that your host’s Ethernet port is connected to bridge br0, if not, edit the interfaces file: 
  $ sudo vi /etc/network/interfaces 
  and replace the eth0 with br0:

   # Replace old eth0 config with br0
   auto  br0
   # Use old eth0 config for br0, plus bridge stuff
   iface br0 inet dhcp
      bridge_ports    eth0
      bridge_stp      off
      bridge_maxwait  0
      bridge_fd       0

- Restart the interface: 
  $ /etc/init.d/networking restart

- Install the bridge-utils and add the new bridge:
  $ sudo apt-get install bridge-utils
  $ sudo brctl addbr br0

- Make sure the bridge is added:
  $ ifconfig

5 - enter the repo directory and start the Qemu VM
$ cd repo
$ sudo ./run

You can login using username "root" and no password

The application is located at /home/root

run "ifconfig" to make sure the eth0 is available and write down the IP address.

You can run the Simple application using the following command:
$ mono SimpleHttpListener.exe

HOWEVER, since we need to run "top" to observe the stats, let's send it to the background:
$ mono SimpleHttpListener.exe >> output.txt &

You can change the protocol to HTTP by modifying the webconfig.json file and restarting the application. You can also change the port to another port, but we have already included a server certificate in the VM which is bounded to port 8002. if you choose another port, you will need to create a new certificate and bind it to the new port.

Once the application runs, you can open a browser on your host machine and access the application at:
http(s)://<VMs_IP_address>:8002/ui/
this simple HTML page keeps making calls to http(s)://<VMs_IP_address>:8002/api/ and prints out a dummy JSON response to the page.

Use "top" and keep an eye on it to observe the CPU and memory usage for HTTPS. The CPU usage is normally around 2% but every 30-40 seconds spikes up to 99%-100% for 10 seconds and goes back down. You can change the protocol to HTTP and re-run the application and observe that there are no high CPU spikes with HTTP.

you can also use:
$ top –d 1 –p <Applications_PID>


Please let me know if I can help in any way,
We really appreciate your help,
Ramtin Kermani
Comment 3 Ramtin Kermani 2016-10-17 17:18:13 UTC
Hello everyone,

Any updates on this? If someone is interested in helping with reproducing the problem and diagnostics, please let me know and we would love to help.

Thanks,
Ramtin
Comment 4 Rodrigo Kumpera 2017-10-11 17:38:48 UTC
Thanks for the repro, we'll take a look at it as we get the time for it.

Note You need to log in before you can comment on or make changes to this bug.