Bug 15451 - HttpWebRequest performs blocking operations exhausting the threadpool when using SSL
Summary: HttpWebRequest performs blocking operations exhausting the threadpool when us...
Alias: None
Product: Class Libraries
Classification: Mono
Component: System ()
Version: master
Hardware: PC Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
Depends on:
Reported: 2013-10-16 21:03 UTC by Bassam
Modified: 2014-02-20 11:33 UTC (History)
2 users (show)

Is this bug a regression?: ---
Last known good build:

Test case (2.43 KB, application/x-gzip)
2013-10-16 21:03 UTC, Bassam

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Bassam 2013-10-16 21:03:50 UTC
Created attachment 5157 [details]
Test case

The attached test case shows that HttpWebRequest performs blocking operations on what should be an async code path when using SSL. The end result is poor performance of SSL conenctions and exhaustion of the threadpool. Many of the stack traces show the following: 

System.Threading.WaitHandle.WaitOne_internal () in 
System.Threading.WaitHandle.WaitOne () in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/corlib/System.Threading/WaitHandle.cs:357
Mono.Security.Protocol.Tls.SslStreamBase.EndNegotiateHandshake (asyncResult={Mono.Security.Protocol.Tls.SslStreamBase.InternalAsyncResult}) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslStreamBase.cs:524
Mono.Security.Protocol.Tls.SslStreamBase.NegotiateHandshake () in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslStreamBase.cs:173
Mono.Security.Protocol.Tls.SslStreamBase.Write (buffer={byte[64]}, offset=0, count=64) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/Mono.Security/Mono.Security.Protocol.Tls/SslStreamBase.cs:1113
System.Net.WebConnection.Write (request={System.Net.HttpWebRequest}, buffer={byte[64]}, offset=0, size=64, err_msg=(null)) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/WebConnection.cs:1151
System.Net.WebConnectionStream.WriteHeaders () in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/WebConnectionStream.cs:679
System.Net.WebConnectionStream.SetHeaders (buffer={byte[64]}) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/WebConnectionStream.cs:649
System.Net.HttpWebRequest.SendRequestHeaders (propagate_error=false) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/HttpWebRequest.cs:1257
System.Net.HttpWebRequest.SetWriteStream (stream={System.Net.WebConnectionStream}) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/HttpWebRequest.cs:1281
System.Net.WebConnection.InitConnection (state={System.Net.HttpWebRequest}) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/WebConnection.cs:787
System.Net.WebConnection.AnonymousMethod__20 (state={System.Net.HttpWebRequest}) in /private/tmp/source/bockbuild-xamarin/profiles/mono-mac-xamarin/build-root/mono-3.2.3/mcs/class/System/System.Net/WebConnection.cs:118

When I run the same test case without SSL it runs really well and does not use much threads.
Comment 1 Sebastien Pouliot 2014-01-25 11:30:50 UTC
Great test case! 100 can be brutal (without [1]) but the effect is still very visible (speed and # of threads [2][3][4]) when using 10 Task.

I'm surprised this did not come up in the iFolder testing. I guess it was because many client (with few connections) were used - so it stayed under the threshold where failures are seen.

[1] https://github.com/mono/mono/pull/829

[2] Running 100 tasks on google.com. Lots of small requests makes the negotiation time very apparent.

new (highest thread id: 30)
	real	0m14.160s
	user	0m19.457s
	sys	0m5.296s

old : all request throwed exceptions (likely due to threadpool exhaustion)
	real	1m48.904s
	user	0m13.142s
	sys	0m3.569s

[3] same as [2] but with 10 tasks.

new (highest thread id: 9)
	real	0m2.728s
	user	0m1.982s
	sys	0m0.331s

old (highest thread id: 23). Note: Two tasks failed (exception)
	real	0m11.953s
	user	0m2.163s
	sys	0m0.436s

[4] Running 10 tasks download each a 1MB file under SSL. The time difference is smaller (%) because the gain (from [1]) is done when negotiating (and it represent a smaller amount of the total SSL connection time). Nevertheless it's a positive gain.

new (highest thread id: 9)
	real	0m24.606s
	user	0m1.970s
	sys	0m0.483s

old (highest thread id: 47)
	real	0m29.550s
	user	0m2.297s
	sys	0m0.777s
Comment 2 Bassam 2014-02-18 12:46:42 UTC
Merged to master 5e03c107bc76e01169bd07cc1a312c73ece59342
Comment 3 Miguel de Icaza [MSFT] 2014-02-20 11:28:10 UTC
Sebastien, while using an old Mono, I do not see any of these exceptions, or a slowdown.   I get timings that are roughly the same.

Do I need to alter the test case?
Comment 4 Sebastien Pouliot 2014-02-20 11:33:03 UTC
Yup, the attached test case use HTTP, not HTTPS

			var req = (HttpWebRequest)HttpWebRequest.Create ("http://www.google.com");

so you need the extra 's' to see the issue.