Created attachment 21876 [details]
When HttpListener is used with SSL, the SSL context and other objects remain in memory causing new memory leak for each accepted connection.
Attached is simple test case - Mono application starting HTTP listener server returning "test" on each request. Then there is a client script - it starts the server with heapshot capability and runs 50 request against it. Then it makes heapshot, generates text file using mprof-report and greps the resulting file for Mono.Btls.MonoBtlsContext. This gives following line:
8000 50 160 Mono.Btls.MonoBtlsContext
So there are 50 instances of context after 50 requests. You can update the script to increase number of connections (iters=50). The test case zip contains also the full text report. The script needs curl and netcat.
The problem is caused by HttpConnection not closing the underlying stream. It just closes the socket but leaves the stream alone. Normally, this is not a problem because it does not keep any system resources and gets garbage collected later when all references are lost.
However, when SSL is used, the stream (MonoBtlsStream) creates various BTLS classes like MonoBtlsContext, MonoBtlsX509Store etc. The problem with these classes is that some of them cooperate with native code and so they allocate GCHandles to themselves to pass to the native code. And these handles are released in Close() methods. So when Close() is not called, all the objects remain in memory even if all references to stream are lost. Because they are effectively not - there are still those GCHandles keeping everything in memory.
I've attached a patch which closes the stream in HttpConnection and thus solves the memory leak.
Created attachment 22331 [details]
Memory leak fix