Bug 53302 - multithread tcp server required time spike
Summary: multithread tcp server required time spike
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: io-layer ()
Version: 4.8.0 (C9)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Ludovic Henry
Depends on:
Reported: 2017-03-14 09:05 UTC by xu lanting
Modified: 2017-03-14 11:45 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

attachment with a performance.jpg and server&client source code (147.93 KB, application/zip)
2017-03-14 09:05 UTC, xu lanting
linux perf report. (771.54 KB, text/plain)
2017-03-14 11:07 UTC, su21
reproduce program (988.05 KB, application/gzip)
2017-03-14 11:45 UTC, su21

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report for Bug 53302 on GitHub or Developer Community if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: GitHub Markdown or Developer Community HTML
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:

Description xu lanting 2017-03-14 09:05:35 UTC
Created attachment 20298 [details]
attachment with a performance.jpg and server&client source code

  I wrote a simple tcp server in C# (run in mono 4.8.0 version), and found some issues in thread management. In my test cases , I opened 100 Goroutines and echoed for 1000 times per Goroutine. However, the results of total latency were not stable, varied from 2s to 13s(as shown in attachment). I found that the requests of thread dispatch were uncommonly high in those bad cases and they took up tons of time. 
  I wonder if there are issues about thread management, which make the thread pool unstable. For example, in some cases, if I started the server and ran the test client for the first time, 24 threads were dispatched. When I ran the    test client second time, it might dispatch more than 35 threads with the same amount of requests. In those 24-thread issues, the latency are low(2s), while others were accordingly high(13s).
Comment 1 su21 2017-03-14 11:04:58 UTC
we observed a lot of spin lock call from work thread which contributes over 20% CPU time in the worst case (according to  Linux perf report)
Comment 2 su21 2017-03-14 11:07:49 UTC
Created attachment 20300 [details]
linux perf report.

produce step:
1.  mono --jitmap ServerProgram
2. perf record -F PID  -p 26092 sleep 30
3. run client multi times.
4. perf script

1. mono --jitmap ServerProgram
2. perf top -p PID
3. run client multi times.
Comment 3 su21 2017-03-14 11:28:22 UTC
try to limit thread pool's thread num via 

System.Threading.ThreadPool.SetMinThreads(5, 5);
System.Threading.ThreadPool.SetMaxThreads(10, 10);
seems like a correct work around.
Process time per round distribute between 1.x second and 5.x second, a lot better.
Comment 4 su21 2017-03-14 11:45:46 UTC
Created attachment 20302 [details]
reproduce program

file list:
csharp server, listen port 6789 by default:
mono TCPServerAsync.exe

run client multi time:
./client -round=10000 -nconn=100