Created attachment 20256 [details]
Reproduces the inability to use JSON.net within a file provider
When I call JsonConvert.DeserializeObject from within my file provider extension, the extension just seems to disappear. I tried using try/catch and that did not catch anything.
I have attached a sample to illustrate the problem.
This issue has me dead in the water at the moment.
I need to be able to download files in my file provider and to do that I make heavy use of JSON.net.
I can reproduce this.
The problem is that there seems to be a 10mb memory limit in the extension, and once that limit is hit, iOS kills your extension.
A quick test with Instruments and a normal iOS app reveals that your simple JSON serialization statement uses ~3mb of memory for a release build (with a debug build and without the debugger attached it's ~4mb; with the debugger attached it would be a bit more, but this is cumbersome to measure with Instruments so I didn't try). Have in mind that most of this memory is allocated because the types and methods in question are used for the first time, if you executed the same statement again this does not indicate how much memory each iteration would need (although Newtonsoft.Json is known for using a significant amount of memory).
I've tested how much memory is available in the extension by allocating memory in small increments and checking when it crashes, and for a release build the extension crashes after allocating ~3.8mb (~2.5mb for a debug build). This means that this is the amount you have to work with (unfortunately the minimal memory footprint in Xamarin.iOS is not as low as we'd wish it to be; this is something we're working on and have plans to improve - see bug #47048 for instance).
My recommendation would be to find an alternative to Newtonsoft.Json to parse JSON (iOS provides the NSJsonSerialization class, but I haven't tried it so I don't know if it might work for you).
How could you tell that iOS was killing my extension? That would be really good to know how to do.
(In reply to tmrog from comment #2)
> How could you tell that iOS was killing my extension? That would be really
> good to know how to do.
There's a crash report called 'JetsamEvent', with a entry whose "reason" field is "per-process-event":
and the "name" field is your executable:
In this case, it's because rpages = 640: https://gist.github.com/rolfbjarne/bd1bea28fc21508ccf6192526dbece19#file-jetsamevent-2017-03-10-140959-ips-L3595, and 640 pages * 16kb per page = 10MB.
Is there a trick to finding these logs? I don't seem to be able to find them in XCode device logs or on my Mac after a sync.
I see that corporate policy has "Don't Send" set for Diagnostics and Usage Data. Does this prevent the log files from being created?
It looks like Xcode does not handle this type of crash reports correctly, Xcode processes it somehow, badly, and it ends up corrupted: https://www.dropbox.com/s/2c48yvnbxs8dj2t/Screenshot%202017-03-14%2011.12.34.png?dl=0
However I was able to get these crash reports by using iTunes, just like Dropbox explains here: https://www.dropbox.com/en/help/5065 (sync device with iTunes, look in ~/Library/Logs/CrashReporter/MobileDevice):
$ grep per-process ~/Library/Logs/CrashReporter/MobileDevice/*/JetsamEvent*.ips -B 1 -A 3
"rpages" : 640,
"reason" : "per-process-limit",
"pid" : 1690,
"idleDelta" : 12892,
"name" : "FileProvider",
Do you have Diagnostics and Usage Data set to automatically send to Apple?
(In reply to tmrog from comment #7)
> Do you have Diagnostics and Usage Data set to automatically send to Apple?
I just tried disabling it, and the JetsamEvent report is still created.
Very strange. Today I am getting them. My rpages is 2560 which is 40MB. I wonder why the difference? I am running iOS 10.2.1 on an iPhone6.
"uuid" : "XXX",
"states" : [
"killDelta" : 5534,
"lifetimeMax" : 5288,
"age" : 2270786529,
"purgeable" : 0,
"fds" : 50,
"genCount" : 0,
"coalition" : 10,
"rpages" : 2560,
"reason" : "per-process-limit",
"pid" : 291,
"idleDelta" : 14320546,
"name" : "FileProvider",
"cpuTime" : 0.562036
My guess would be that the page size is 4kb instead of 16kb (so that would be 2560 pages * 4kb = 10mb). In the end the memory limit is the same.
One potential reason is that you're running in a 32-bit process instead of 64-bit process.
You are right the page size 4kb. Are settings wrong in my FileProvider project or is it dependent on the host app?
BTW...I found this JSON library which doesn't seem to kill the FileProvider. It doesn't handle things as seamlessly as JSON.net and I have had to modify it for parsing DateTime. Also, doesn't seem to handle enums so I am trying to figure out how to make that work.
(In reply to tmrog from comment #11)
> You are right the page size 4kb. Are settings wrong in my FileProvider
> project or is it dependent on the host app?
A 4kb page size is completely normal (iOS decides it), you only have to be aware if you want to calculate the memory limit from the rpages value in the crash report.
So, I got SimpleJson working but now have just kicked the problem down the road. When I try to make a web request using HttpClient and HttpClientHandler my file provider gets terminated.
1. It is my understanding the NSUrlSessionHandler uses less memory. I would love to use that but cannot due to: https://bugzilla.xamarin.com/show_bug.cgi?id=53220. Any ideas?
2. Any other way to get my memory footprint down?
3. Is it possible to write my FileProvider with Xcode and provide it with my Xamarin app?
(In reply to tmrog from comment #13)
> So, I got SimpleJson working but now have just kicked the problem down the
> road. When I try to make a web request using HttpClient and
> HttpClientHandler my file provider gets terminated.
> 1. It is my understanding the NSUrlSessionHandler uses less memory. I would
> love to use that but cannot due to:
> https://bugzilla.xamarin.com/show_bug.cgi?id=53220. Any ideas?
You can use the iOS API instead of the managed API (NSUrlSession for instance: https://developer.xamarin.com/api/type/Foundation.NSUrlSession/)
> 2. Any other way to get my memory footprint down?
> 3. Is it possible to write my FileProvider with Xcode and provide it with my
> Xamarin app?
Yes, this is possible, but not trivial.
I helped another customer do this here: https://bugzilla.xamarin.com/show_bug.cgi?id=43985#c12, try following those steps, and just ask (in this bug) if you run into problems and I'll help you.
I finally got my file provider working by doing as you said and using NSUrlSession directly. That allowed me to make web requests. The final piece was to use a download task instead of the data task for the file download. Sounds, of course, obvious but initially just used the same code I had written for making other requests.
Is there a way for me to programmatically determine how much memory I am using? Hard to monitor it in other ways as the File Provider seems to come and go so quickly.
Also, the only assemblies being loaded now are below. Are those the minimum assuming I need the Plugin.Settings package?
Loaded assembly: /.monotouch-64/System.dll [External]
Loaded assembly: /.monotouch-64/Xamarin.iOS.dll [External]
Loaded assembly: /.monotouch-64/Mono.Dynamic.Interpreter.dll [External]
Loaded assembly: /.monotouch-64/System.Core.dll [External]
Loaded assembly: /.monotouch-64/Plugin.Settings.Abstractions.dll [External]
Loaded assembly: /.monotouch-64/Plugin.Settings.dll [External]
Loaded assembly: /.monotouch-64/FileProvider.dll
(In reply to tmrog from comment #15)
> Also, the only assemblies being loaded now are below. Are those the minimum
> assuming I need the Plugin.Settings package?
> Loaded assembly: /.monotouch-64/System.dll [External]
> Loaded assembly: /.monotouch-64/Xamarin.iOS.dll [External]
> Loaded assembly: /.monotouch-64/Mono.Dynamic.Interpreter.dll [External]
> Loaded assembly: /.monotouch-64/System.Core.dll [External]
> Loaded assembly: /.monotouch-64/Plugin.Settings.Abstractions.dll [External]
> Loaded assembly: /.monotouch-64/Plugin.Settings.dll [External]
> Loaded assembly: /.monotouch-64/FileProvider.dll
This looks good; but if curious you can look at each assembly's dependencies by using the monodis tool:
$ monodis --assemblyref /path/to/System.dll
0x00000000: 7C EC 85 D7 BE A7 79 8E
0x00000000: 7C EC 85 D7 BE A7 79 8E
This shows that System.dll references mscorlib.dll and System.Xml.dll.
Thanks for all your help on this. I guess in summary for anybody who reads this bug, I had to do the following things to get my file provider extension to work:
1. Remove Json.net from the project.
2. Used simple-json instead - https://github.com/facebook-csharp-sdk/simple-json
3. Stop using managed HTTP layer. Remove System.Net.Http from the project
4. Used NSUrlSession directly.
5. Moral of the story is, you must use as little memory as possible. Reduce the assemblies you use in addition to any other memory consumption.
*** Bug 58167 has been marked as a duplicate of this bug. ***
Now that bug #56022 has been fixed, memory usage has decreased, but it doesn't look like enough to make JSON.Net work.
It's entirely possible that JSON.Net will never work, if it allocates more than iOS allows for file provider extensions there's nothing we can do about it.
I'm resolving this as fixed, since the reporter was able to make it work according to comment #17.