Bug 386 - Memory leak using WCF
Summary: Memory leak using WCF
Status: REOPENED
Alias: None
Product: Class Libraries
Classification: Mono
Component: WCF assemblies (show other bugs)
Version: unspecified
Hardware: All All
: Highest critical
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on: 2619 2620
Blocks:
  Show dependency tree
 
Reported: 2011-08-24 16:23 UTC by Louis Boux
Modified: 2016-06-29 09:37 UTC (History)
12 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
projects and screenshots (1.68 MB, application/zip)
2011-08-24 16:23 UTC, Louis Boux
Details
After clicking once on the 100mb button, the app receives 100mb from the server (207.34 KB, image/png)
2011-08-26 11:22 UTC, Louis Boux
Details
At the same time, the activity monitor shows that the app takes way over 1GB of ram (176.30 KB, image/png)
2011-08-26 11:23 UTC, Louis Boux
Details
StringBuilder memory leak projects and screenshots (1.78 MB, application/zip)
2011-09-28 14:06 UTC, Louis Boux
Details

Description Louis Boux 2011-08-24 16:23:12 UTC
Created attachment 181 [details]
projects and screenshots

I have a WCF service running on a windows machine and an iPad client getting byte arrays from it. In the attachment, I have included the server project (created in Visual Studio 2010) and the iPad client project. After receiving a byte array of 100 MB, the amount of RAM used by the app skyrockets over 1 GB and stays up there. It does go down a little but never close to where it was before receiving the byte array from the server.

== READ ME.txt contents ==
1. Run the Server on a windows platform (inside TestTouchService.zip, which is a project created using Visual Studio 2010), which will start the service on 127.0.0.1:9999.

2. In the TestTouchServiceClient project, which was created as an iPad project using MonoDevelop, change the value of the TestTouchClient.Address constant to the ip of the machine currently hosting the service.

3. Run the TestTouchServiceClient, tap any button to get the number of bytes indicated on them from the server. If it crashes on boot, it probably just needs a clean and rebuild.

4. If you are running it on the device, the app will tell you when you are low on memory and in the next seconds your app is very likely to crash. When i tested on my iPad 2, the app crashed after receiving a little more than 42 MB, but I suggest running it on the simulator, so you can see how much memory it takes from the Activity Monitor. After pressing the 100 MB button, the app takes over 1 GB of RAM, as shown in the screenshots.
Comment 1 Louis Boux 2011-08-26 11:22:25 UTC
Created attachment 196 [details]
After clicking once on the 100mb button, the app receives 100mb from the server
Comment 2 Louis Boux 2011-08-26 11:23:08 UTC
Created attachment 197 [details]
At the same time, the activity monitor shows that the app takes way over 1GB of ram
Comment 3 Louis Boux 2011-08-26 13:21:16 UTC
forgot to say I am using Snow Leopard (10.6.6), not Lion
Comment 4 Sebastien Pouliot 2011-08-29 21:31:00 UTC
I did a quick check (using 10MB chunks) and yes memory goes way up and never down as much as expected.

Simulator (in MB)
              up to         stable
initial                       34.9
10MB      163.1        85.2
20MB      191.9        128.0
30MB      242.3        169.1
40MB      262.8        210.2
50MB      324.5        226.2
60MB      324.6*      251.3
70MB      392.4        251.3
80MB      417.9        319.1
90MB      442.7        360.1
100MB    485.1        401.2

I'm not sure if it's a "real" leak or if some things never gets unreferenced (e.g. cache). Hopefully we'll soon have better tools (e.g. heapshot) to diagnose those. OTOH a console application would likely show the same behavior.
Comment 5 Jin Lee 2011-09-28 10:23:19 UTC
We happen to have the same problem for our app which uses WCF communication.  When dealing with significant chunks through the wire memory just sky rockets both on simulator and on the device. 

Even though its only a specific part of our app that is affected by the issue (small chunks never seem to cause the issue), we are feeling uneasy about releasing knowing that problem, any news on this one ?
Comment 6 Louis Boux 2011-09-28 14:05:43 UTC
Update : Recently I had another problem with a very long string built using StringBuilder ; Memory was allocated and never totally released, very similar to the problem I have been describing in this bug case. If MonoTouch is using StringBuilder in its WCF Deserialization, I guess it would make sense, because deserializing a few hundred MB using StringBuilder would most likely display the same behavior.

After some research, I also found an already existing question on StackOverflow describing this issue, but the answer provided no functional solution.

see here : http://stackoverflow.com/questions/4997276/why-is-this-code-crashing-an-ios-app-with-out-of-memory

I am also joining a sample project (screenshots included) that I used to confirm the StringBuilder bug. I suspect that StringBuilder's implementation when faced with very large strings has a problem with heap fragmentation and subsequent memory de-allocation, which would lead to the leak (for a high amount of small strings built, there's no problem).

I hope this will help resolve the issue quickly.
Comment 7 Louis Boux 2011-09-28 14:06:41 UTC
Created attachment 524 [details]
StringBuilder memory leak projects and screenshots
Comment 8 Miguel de Icaza [MSFT] 2011-09-28 14:29:25 UTC
A console version of the same program, exhibiting the same issue:


using System.Text;
using System;

class X {
	static void Main ()
	{
		while (true){
			Console.WriteLine ("Press return");
			Console.ReadLine ();
			Alloc ();
		}
	}

	static void Alloc ()
	{
            var s = new StringBuilder();

            // append lots of times
            for (int j = 0; j < 1024 * 1024 * 200; j++)
                s.Append('A');
	    s.Length = 0;
	    s = null;
	}
}
Comment 9 Paolo Molaro 2011-09-29 05:00:52 UTC
This is the effect of us using some amount of conservative pointer scanning in the GC. A Typical 32 bit app has, say, 2 GB of address space available for allocations and the sample code above allocated 400+MB objects, so just 5 of them are enough to fill the memory space in the unfortunate case where we find a value on the stack that looks like a pointer to the area used by the object.
So this has nothing to do with the StringBuilder implementation specifically, as it can happen with any array that is just as large.
As a proof, remove the "* 200" bit and the app will run indefinitely (well, I stopped it after a few minutes here, but it didn't increase memory used). The original program likely runs fine as is on a 64 bit architecture.

As for a solution:
*) for the case of WCF the implementation shouldn't use StringBuilders if the amount of memory it could
retain is unbounded. Chunked byte[] buffers is likely the best solution there (though I didn't look at the implementation and the API requirements). Alternatively, try to avoid hundreds of megabyte transfers at the app level (it may or may not be possible, depending on who controls the two ends of the interfaces).
*) for direct uses of StringBuilder my advice is basically the same: it is not really suitable to manage really large strings. Think of it as processing XML files: it is a bad idea to use a parser that requires to keep all the content in memory if the input file size is unbounded, better use something like XmlReader.

It has been suggested that StringBuilder could internally use a chunked implementation: my opinion is that this is a bad idea, since it would slow down significantly the implementation for the common case to cover a corner case for which StringBuilder is not the right solution anyway.
Comment 10 Louis Boux 2011-10-03 10:35:15 UTC
I mainly used the StringBuilder as an example to represent the problem, the issue with WCF is much more pressing. If it's true that it is an issue related to large arrays, it is much more troublesome than I thought, as it brings out several other cases in our project where that may apply, although not as critical (a decent amount of those can be solved client-side).

Most of the data transfers in our project are actually of very reasonable sizes, with some corner cases around pictures, audio, and video that are our heavyweights, of those some can and will be chunked/streamed, some can't. 

What worries me is that based on what you said, the moment a large array allocation appears, the app technically won't ever be able to release most of the memory allocated for it, and will be stuck with it. Thus, multiple instances of such an event without the app closing will inevitably result in an "out of memory" crash.

I think that chunking will unlikely solve problems in big transfer cases anyway. Yes, data will be received in managable chunks and deserialization will not leak, but most of the time there will be a need to merge it all back together eventually in a single array in order to actually consume the data received, at which point the array size problem kicks in regardless.

I suppose if we had any way to manually release the memory allocated for unused arrays it could probably work. Anyhow, do you guys have any other suggestion to solve this problem, at least temporarily, as it could very well delay our app submission in a significant manner.
Comment 11 Miguel de Icaza [MSFT] 2011-10-03 11:30:09 UTC
We are going to look at the WCF problem and see if there is something we can do about it.

Meanwhile, for transferring images, you could use the plain HTTP stack and store the results on disk, instead of keeping those in memory and deserializing it (audio, video and pictures)
Comment 12 Rolf Bjarne Kvinge [MSFT] 2011-11-30 17:28:30 UTC
=> WCF
Comment 13 Rolf Bjarne Kvinge [MSFT] 2011-12-02 06:35:43 UTC
Louis, the recently released 5.1.0 beta has significantly improved the support for our new garbage collector (SGen), which should in theory resolve your issues. Can you try it to see if it actually does?
Comment 14 Louis Boux 2011-12-05 14:32:23 UTC
I just tested it with 5.1.0 beta, and it's still exactly the same. I received 10 MB through WFC and it skyrocketed (to around 161 MB) before falling back down a little but nowhere near what it was before receiving the 10 MB.
Comment 15 Rolf Bjarne Kvinge [MSFT] 2011-12-05 17:05:56 UTC
Louis, did you try enabling SGen in the project's properties?
Comment 16 Louis Boux 2011-12-06 10:34:22 UTC
That's probably a good idea.

Results after enabling SGen : 
The memory still skyrockets in the same way, but it does go back down more than without SGen, though still not completely.

a few numbers :
if I just keep receiving chunks of 10 MB one after another, the mem goes up to 130 MB and stays there, If I then receive chunks of 5 MB or 1 MB, the memory goes back down under 130 MB (around 50 and 80 MB), but as soon as I go over 10 MB, such as 25 MB, the memory gets stuck at around 200 MB, and if I do it again, it just gets worst.

So my guess is that what enabling SGen did in this case was to rise the threshold of the byte array's size where it goes completely crazy.

Even though it's unreal to receive 25 MB through WCF in a real application, I still feel like I'm playing with explosives.

I guess it's safe for now..... (famous last words)
Comment 17 Rolf Bjarne Kvinge [MSFT] 2011-12-16 21:22:23 UTC
I can reproduce the memory graph you're seeing with sgen. I profiled it somewhat, but didn't find the cause of it (yet).
Comment 18 Rolf Bjarne Kvinge [MSFT] 2011-12-22 18:30:00 UTC
I found one bug in the threadpool, opened case #2619 for it. It looks like there is at least one other bug somewhere too.
Comment 19 Rolf Bjarne Kvinge [MSFT] 2011-12-22 19:07:09 UTC
Filed another bug for the second leak: #2620.
Comment 20 Rolf Bjarne Kvinge [MSFT] 2012-01-17 11:48:23 UTC
Both bug #2619 and bug #2620 have been fixed now. The fixes will be included in MonoTouch 5.2.
Comment 21 Cody Beyer (MSFT) 2014-12-12 11:46:29 UTC
Issue is occurring again using the same test case as noted above
Comment 23 Cody Beyer (MSFT) 2014-12-12 12:07:08 UTC
Xamarin Studio
Version 5.5.4 (build 15)
Installation UUID: 63e65309-d2a1-425e-a033-2dbdf229f771
Runtime:
	Mono 3.10.0 ((detached/92c4884)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 310000031

Apple Developer Tools
Xcode 6.1 (6602)
Build 6A1052c

Xamarin.Android
Not Installed

Xamarin.iOS
Version: 8.4.0.43 (Business Edition)
Hash: 840a925
Branch: 
Build date: 2014-11-16 21:03:22-0500

Xamarin.Mac
Version:

Build Information
Release ID: 505040015
Git revision: f93940a35458a18052f1a25e106e62ca970d9c40
Build date: 2014-11-19 15:32:41-05
Xamarin addins: dc23cbd91a3a0e1d326328e1229e86c942a49ec8

Operating System
Mac OS X 10.10.1

Note You need to log in before you can comment on or make changes to this bug.