*** Summary of the issue ***
Initially, running a simple self-hosting OWIN application that serves HTTP(S) endpoints, we observed (when using HttpS vs Http) the CPU usage on the device spikes up to 100% every 30 seconds, stays up for about 10 seconds and then falls back to about 10% which is the normal amount. The simple application serves an HTML page that makes ajax calls to the REST endpoints exposed by the application and pulls some data every 2-3 seconds. The same application does not behave this way when we switch to HTTP (instead of HTTPS) and there are no spikes. To determine if Mono implementation of HTTPS is the problem or the OWIN/WebAPI, we eliminated OWIN and created a "Simple HTTP listener" that simply listens to an Http(S) connection and serves the endpoints. We again observed the issue. Note that this problem does not occur on a 64 bit desktop Ubuntu machine with 4 cores.
*** System Characteristics ***
- An embedded device running a custom Yocto Linux
- Application runs inside an LXC container which is hosted inside a Linux VM
- Single core 1.25 GHz 32 bit Atom CPU
- 360 MB of RAM
- Mono Version: 22.214.171.124
*** Reproducing the issue ***
We have created a simple application that you can build and run on a "preferably" single core Linux device (rather than a 64 bit desktop Linux). You may emulate the device using something like Qemu. The application code is placed at Github: https://github.com/ramtinkermani/SimpleHttpListener-CSharp
*** Configuring the application ***
- You can configure the HTTP(S) port and the protocol (HTTP vs HTTPS) in the webconfig.json file. The host name is irrelevant here.
- If using HTTPS, you need to create a self-signed Server certificate for the server.
- Also Mono implementation of the HTTPS requests the Client to send a (Any!) Client certificates to the server in order to establish a secure communication. So you need to either create a Client certificate and import it into your browser or use an existing Client Certificate that is already available in your browser.
*** Profiling ***
To observe the spikes, once you run the application you can find the PID of the process using the following command:
$ pgrep mono and then use that PID in the following command to print out the CPU/Memory usage. You can either write that data to a file for a couple of minutes or simply keep an eye on the numbers to observe the CPU spiking up to 100% every ~30 seconds.
$ top -p <PID>
Given that this only occurs on a very specific hardware/OS configuration that we don't have available and can't easily get access to, we really need more specific instructions on how to reproduce this under something like qemu. What exact options should we use to match as closely as possible your actual hardware? What OS and userland should we use?
Since this is a bug that only occurs on very niche hardware, it's hard for us to justify spending a ton of time trying to replicate your environment. If you can give us clear instructions on how to do so, though, it would be much easier for us to actually debug and fix the issue.
Additionally/alternatively, if you're feeling adventurous, you could try AOTing all assemblies in your application and running Mono with the `perf` tool. This could give us some idea of where all this CPU time is being spent. (I would normally suggest using the Mono profiler, but 4.2.1 unfortunately has a lot of stability bugs in the profiler that have since been fixed. You could still give it a shot, though.)
We created a Qemu image that to a good extent emulates our device. We have included the Simple application. Here are the steps to reproduce the issue and investigate the performance:
1- Download the Qemu image from the following address:
which contains :
- The Kernel
- root filesystem
- run script (A one-liner script to avoid typing the long qemu run command!)
- Mono is already installed and in Path
- Simple application
2- Uncompress the tarball:
$ tar xvfz QemuVM.tr.gz
3- make sure Qemu is installed on your machine (Mine is a 32bit Ubuntu 14.04)
$ sudo apt-get install qemu-system-x86
4- It is assumed that your host’s Ethernet port is connected to bridge br0, if not, edit the interfaces file:
$ sudo vi /etc/network/interfaces
and replace the eth0 with br0:
# Replace old eth0 config with br0
# Use old eth0 config for br0, plus bridge stuff
iface br0 inet dhcp
- Restart the interface:
$ /etc/init.d/networking restart
- Install the bridge-utils and add the new bridge:
$ sudo apt-get install bridge-utils
$ sudo brctl addbr br0
- Make sure the bridge is added:
5 - enter the repo directory and start the Qemu VM
$ cd repo
$ sudo ./run
You can login using username "root" and no password
The application is located at /home/root
run "ifconfig" to make sure the eth0 is available and write down the IP address.
You can run the Simple application using the following command:
$ mono SimpleHttpListener.exe
HOWEVER, since we need to run "top" to observe the stats, let's send it to the background:
$ mono SimpleHttpListener.exe >> output.txt &
You can change the protocol to HTTP by modifying the webconfig.json file and restarting the application. You can also change the port to another port, but we have already included a server certificate in the VM which is bounded to port 8002. if you choose another port, you will need to create a new certificate and bind it to the new port.
Once the application runs, you can open a browser on your host machine and access the application at:
this simple HTML page keeps making calls to http(s)://<VMs_IP_address>:8002/api/ and prints out a dummy JSON response to the page.
Use "top" and keep an eye on it to observe the CPU and memory usage for HTTPS. The CPU usage is normally around 2% but every 30-40 seconds spikes up to 99%-100% for 10 seconds and goes back down. You can change the protocol to HTTP and re-run the application and observe that there are no high CPU spikes with HTTP.
you can also use:
$ top –d 1 –p <Applications_PID>
Please let me know if I can help in any way,
We really appreciate your help,
Any updates on this? If someone is interested in helping with reproducing the problem and diagnostics, please let me know and we would love to help.
Thanks for the repro, we'll take a look at it as we get the time for it.