Bug 53302 - multithread tcp server required time spike
Summary: multithread tcp server required time spike
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: io-layer (show other bugs)
Version: 4.8.0 (C9)
Hardware: PC Linux
: --- normal
Target Milestone: ---
Assignee: Ludovic Henry
Depends on:
Reported: 2017-03-14 09:05 UTC by xu lanting
Modified: 2017-03-14 11:45 UTC (History)
4 users (show)

See Also:
Is this bug a regression?: ---
Last known good build:

attachment with a performance.jpg and server&client source code (147.93 KB, application/zip)
2017-03-14 09:05 UTC, xu lanting
linux perf report. (771.54 KB, text/plain)
2017-03-14 11:07 UTC, su21
reproduce program (988.05 KB, application/gzip)
2017-03-14 11:45 UTC, su21

Description xu lanting 2017-03-14 09:05:35 UTC
Created attachment 20298 [details]
attachment with a performance.jpg and server&client source code

  I wrote a simple tcp server in C# (run in mono 4.8.0 version), and found some issues in thread management. In my test cases , I opened 100 Goroutines and echoed for 1000 times per Goroutine. However, the results of total latency were not stable, varied from 2s to 13s(as shown in attachment). I found that the requests of thread dispatch were uncommonly high in those bad cases and they took up tons of time. 
  I wonder if there are issues about thread management, which make the thread pool unstable. For example, in some cases, if I started the server and ran the test client for the first time, 24 threads were dispatched. When I ran the    test client second time, it might dispatch more than 35 threads with the same amount of requests. In those 24-thread issues, the latency are low(2s), while others were accordingly high(13s).
Comment 1 su21 2017-03-14 11:04:58 UTC
we observed a lot of spin lock call from work thread which contributes over 20% CPU time in the worst case (according to  Linux perf report)
Comment 2 su21 2017-03-14 11:07:49 UTC
Created attachment 20300 [details]
linux perf report.

produce step:
1.  mono --jitmap ServerProgram
2. perf record -F PID  -p 26092 sleep 30
3. run client multi times.
4. perf script

1. mono --jitmap ServerProgram
2. perf top -p PID
3. run client multi times.
Comment 3 su21 2017-03-14 11:28:22 UTC
try to limit thread pool's thread num via 

System.Threading.ThreadPool.SetMinThreads(5, 5);
System.Threading.ThreadPool.SetMaxThreads(10, 10);
seems like a correct work around.
Process time per round distribute between 1.x second and 5.x second, a lot better.
Comment 4 su21 2017-03-14 11:45:46 UTC
Created attachment 20302 [details]
reproduce program

file list:
csharp server, listen port 6789 by default:
mono TCPServerAsync.exe

run client multi time:
./client -round=10000 -nconn=100

Note You need to log in before you can comment on or make changes to this bug.