Bug 8284 - Memory leak in mod_mono
Summary: Memory leak in mod_mono
Status: NEW
Alias: None
Product: Class Libraries
Classification: Mono
Component: System.Web (show other bugs)
Version: unspecified
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2012-11-08 12:35 UTC by Ben
Modified: 2015-09-21 11:05 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
profile output at 10MB memory and ~2000 hits. (6.23 MB, application/octet-stream)
2012-11-09 07:01 UTC, Ben
Details
Profile #2 (1.13 MB, application/octet-stream)
2012-11-09 10:25 UTC, Ben
Details

Description Ben 2012-11-08 12:35:36 UTC
I have found that calling a webservice method within Apache2 and mod_mono causes a predictable memory leak.  On a popular server this will continue until some memory limit it hit, at which time mod_mono will crash.

Using mono 3.0.0, mod_mono 2.10, xsp 2.10, SUSE 12.1, Linux 3.1.10 x86_64.

Create a simple Webservice method:

  [WebMethod]
  public void Hit() {
    iHits++;
  }

Call this continually whilst monitoring the memory to a log file via the call:

  GC.GetTotalMemory(false)

I tested this at about 44.8 hits/second in parallel from a collection of 10 threads from a client application.

In 1140 seconds, this consumed 27MB of lost memory, or 542 byes per hit.

This memory was not recovered after the test completed, even after waiting quite some time.

As stated: this will continue until the server hits some memory limit, and will then crash.  Example exception after allocation of 8GB limit:

Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Stacktrace:

  at <unknown> <0xffffffff>
  at (wrapper managed-to-native) object.__icall_wrapper_mono_string_alloc (intptr) <0xffffffff>
  at (wrapper alloc) object.Alloc (intptr,int) <0xffffffff>
  at string.Concat (string,string) <0x00079>
  at string.Concat (object,object) <0x0005b>
  at ...


I have tried both Boehm and SGen.  The result is the same in both.

I have attached an output.mlpd for a ~20,000 hit run.  I am no expert on reading these, but I can see the object 'System.Net.Sockets.Socket.Worker' listed about the same number of times as the hits.  However I may be miss-reading the data.
Comment 1 Ben 2012-11-08 12:48:31 UTC
Sorry I can't attach the memory dump in output.mlpd, file too big.  I will email it on request.
Comment 2 Zoltan Varga 2012-11-08 13:09:34 UTC
I don't know too much about mod-mono, so I'm not sure I can help. There leak can be either in managed code, or in the mono runtime, or in mod_mono. For the first case, try running with sgen with a command line like this:

/mono-sgen --profile=log:heapshot=10gc,output=-x.log,nocalls

This will print the contents of the heap into x.log for every 10th garbage collection. Running
mprof-report x.log
will print out the contents of the heap, allowing you to see which objects consume the most memory. Hopefully these will be the leaking objects.
Comment 3 Ben 2012-11-09 05:19:49 UTC
Hi Zoltan,

Thanks for the suggestion.  I am really no expert in understanding these, however the object 'System.Net.Sockets.Socket.Worker' appears in the log at about the same count as the hits to the server.   If this object is not being (i) dereference by mod_mono_server application or (ii) released by SGen: this might be the cause.  (I have heard the the GC has a problem with steams?  This is a stream?)

However that's my assumption.  I am hoping that some more learned programmer with familiarity with mod_mono_server2 might have be able to give a more accurate answer...
Comment 4 Zoltan Varga 2012-11-09 05:50:19 UTC
Can you attach the log file created by the run ? It should be much smaller than the previous one.
Comment 5 Ben 2012-11-09 07:00:04 UTC
As the file grows in size fast, I was only able to get test data for a short period.  Terminated when GC.GetTotalMemory(false) reported 10MB memory allocated.  This represented ~2200 hits passed to mod-mono-server2.

The OS recorded this memory footprint:

VmPeak:   819468 kB
VmSize:   757644 kB
VmLck:         0 kB
VmHWM:     42800 kB
VmRSS:     42800 kB
VmData:   717232 kB
VmStk:       136 kB
VmExe:      3324 kB
VmLib:      5656 kB
VmPTE:       268 kB
VmSwap:        0 kB
Threads:        12

The end of the report shows:

Heap shot 0 at 0.010 secs: size: 2720, object count: 25, class count: 7, roots: 0
             Bytes      Count  Average Class name
              1408          2      704 System.Object[]
               600         15       40 System.MonoType
               240          2      120 System.Byte[]
               168          1      168 System.AppDomainSetup
               144          1      144 System.AppDomain
                80          2       40 System.MonoListItem
                80          2       40 System.MonoCQItem

Full file attached.
Comment 6 Ben 2012-11-09 07:01:42 UTC
Created attachment 2890 [details]
profile output at 10MB memory and ~2000 hits.
Comment 7 Zoltan Varga 2012-11-09 09:03:02 UTC
Thats only the heap state after the first GC. Could you run it longer, and run
mprof-report on the log file, and attach the output of that ?
Comment 8 Ben 2012-11-09 10:24:35 UTC
Trying again...

Running time: ~14 mins
Hits on mod-mono-server2: ~56000
Memory from GC.GetTotalMemory(false): 31649 KB
Memory from OS: 790348 KB
GC runs from GC.CollectionCount(0): 1031
GC runs from GC.CollectionCount(1): 5
Size of x.log: 1183391 B (Attached)

End of mprof-report x.log:

Heap shot 0 at 0.007 secs: size: 2720, object count: 25, class count: 7, roots: 0
             Bytes      Count  Average Class name
              1408          2      704 System.Object[]
               600         15       40 System.MonoType
               240          2      120 System.Byte[]
               168          1      168 System.AppDomainSetup
               144          1      144 System.AppDomain
                80          2       40 System.MonoListItem
                80          2       40 System.MonoCQItem

I am sorry, not getting anything else.

The directive 'heapshot=10gc', is that for generation 0 or 1?
Comment 9 Ben 2012-11-09 10:25:54 UTC
Created attachment 2891 [details]
Profile #2
Comment 10 Ben 2012-11-09 10:53:25 UTC
One last attempt...  A big run.

Hits on mod-mono-server2: ~515000
Memory from GC.GetTotalMemory(false): 217137 KB
Memory from OS: 1105664 KB
GC runs from GC.CollectionCount(0): 9119
GC runs from GC.CollectionCount(1): 16
Size of x.log: 8654 MB (Not Attached)

End of mprof-report x.log:

 Heap shot 547 at 520.244 secs: size: 96940536, object count: 1163139, class count: 640, roots: 0
             Bytes      Count  Average Class name
          22036592     371913       59 System.Object[] (bytes: +11208, count: +153)
          19906152     355467       56 System.Collections.Queue (bytes: +6272, count: +112)
          15640504     177733       88 System.Net.Sockets.Socket (bytes: +4928, count: +56)
           8830024       7195     1227 System.Byte[] (bytes: +11264, count: +8)
           7257048      96253       75 System.String (bytes: +11600, count: +157)
           3531992       2453     1439 System.Collections.Generic.Link[] (bytes: +1056, count: +4)
           2881336          1  2881336 System.Net.Sockets.Socket[] (bytes: +0, count: +0)
           2725976      30977       88 System.Web.IntPtrStream (bytes: +5016, count: +57)
           2427200       8763      276 System.Int32[] (bytes: +1904, count: +18)
           1616184       9461      170 System.String[] (bytes: +2800, count: +18)
           1330288       5187      256 System.Collections.Hashtable.Slot[] (bytes: +2728, count: +11)
            933480       7779      120 System.EventHandler (bytes: +480, count: +4)
            621480       5179      120 System.Collections.Hashtable (bytes: +1320, count: +11)
            620032      15501       39 System.Collections.ArrayList (bytes: +1560, count: +39)
            561960       1115      504 Mono.WebServer.ModMonoWorkerRequest (bytes: +1104, count: +2)
            558440      13961       40 System.Collections.Specialized.NameObjectCollectionBase._Item (bytes: +1320, count: +33)
            494448       3866      127 System.Char[] (bytes: +272, count: +6)
            392480       1115      352 System.Web.HttpRequest (bytes: +736, count: +2)
            360336          4    90084 System.Boolean[] (bytes: +0, count: +0)
            278200       1122      247 Mono.WebServer.ModMonoRequest (bytes: +552, count: +2)
            276520       1115      248 System.Web.HttpResponse (bytes: +496, count: +2)
            205480       2335       88 System.Collections.Generic.Dictionary<System.String,System.String> (bytes: +352, count: +4)
            202928       2306       88 System.IO.MemoryStream (bytes: +352, count: +4)
            196240       1115      176 System.Web.HeadersCollection (bytes: +528, count: +3)
            196240       1115      176 System.Web.HttpContext (bytes: +352, count: +2)
            196240       1115      176 System.Uri (bytes: +528, count: +3)
            169536       3760       45 System.Delegate[] (bytes: +0, count: +0)
            163200        120     1360 System.Xml.NameTable.Entry[] (bytes: +0, count: +0)
            133800       1115      120 System.Web.HttpHeaderCollection (bytes: +360, count: +3)
            133800       1115      120 System.Web.HttpResponseStream (bytes: +240, count: +2)

I hope that helps....
Comment 11 Zoltan Varga 2012-11-09 12:50:49 UTC
This trace is useful. Moving to class libs/Sys.Web.
Comment 12 Béla Klotz 2013-06-10 07:24:48 UTC
Hello,

what is the status of this problem? (we face similar leaking with mono 2.10.9 and mod-mono-server4 2.10)

thanx
Comment 13 Béla Klotz 2014-05-22 03:51:21 UTC
the problem is in socket handling within xsp component
it does not cleanup the 
                Dictionary <Socket, bool> registeredSockets;
which is only used to cleanup at shutdown, so
after commenting out the xsp-2.10/src/Mono.WebServer/ApplicationServer.cs line 487:
                        //RegisterSocket (accepted);
no memory leak was present.
The functionality is not affected by this patch for mod_mono.exe, but maybe has side effects for xsp.exe

Note You need to log in before you can comment on or make changes to this bug.