Bug 53207 - Can't use JSON.net in File Provider Extension
Summary: Can't use JSON.net in File Provider Extension
Status: ASSIGNED
Alias: None
Product: iOS
Classification: Xamarin
Component: General (show other bugs)
Version: XI 10.4 (C9)
Hardware: PC Mac OS
: --- normal
Target Milestone: Future Cycle (TBD)
Assignee: Rolf Bjarne Kvinge [MSFT]
URL:
: 58167 (view as bug list)
Depends on: 56022
Blocks:
  Show dependency tree
 
Reported: 2017-03-09 20:20 UTC by tmrog
Modified: 2017-08-04 14:53 UTC (History)
3 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Reproduces the inability to use JSON.net within a file provider (4.16 MB, application/zip)
2017-03-09 20:20 UTC, tmrog
Details

Description tmrog 2017-03-09 20:20:40 UTC
Created attachment 20256 [details]
Reproduces the inability to use JSON.net within a file provider

When I call JsonConvert.DeserializeObject from within my file provider extension, the extension just seems to disappear.  I tried using try/catch and that did not catch anything.

I have attached a sample to illustrate the problem.  

This issue has me dead in the water at the moment.

I need to be able to download files in my file provider and to do that I make heavy use of JSON.net.
Comment 1 Rolf Bjarne Kvinge [MSFT] 2017-03-10 16:16:57 UTC
I can reproduce this.

The problem is that there seems to be a 10mb memory limit in the extension, and once that limit is hit, iOS kills your extension.

A quick test with Instruments and a normal iOS app reveals that your simple JSON serialization statement uses ~3mb of memory for a release build (with a debug build and without the debugger attached it's ~4mb; with the debugger attached it would be a bit more, but this is cumbersome to measure with Instruments so I didn't try). Have in mind that most of this memory is allocated because the types and methods in question are used for the first time, if you executed the same statement again this does not indicate how much memory each iteration would need (although Newtonsoft.Json is known for using a significant amount of memory).

I've tested how much memory is available in the extension by allocating memory in small increments and checking when it crashes, and for a release build the extension crashes after allocating ~3.8mb (~2.5mb for a debug build). This means that this is the amount you have to work with (unfortunately the minimal memory footprint in Xamarin.iOS is not as low as we'd wish it to be; this is something we're working on and have plans to improve - see bug #47048 for instance).

My recommendation would be to find an alternative to Newtonsoft.Json to parse JSON (iOS provides the NSJsonSerialization class, but I haven't tried it so I don't know if it might work for you).
Comment 2 tmrog 2017-03-10 16:59:41 UTC
How could you tell that iOS was killing my extension?  That would be really good to know how to do.
Comment 3 Rolf Bjarne Kvinge [MSFT] 2017-03-13 07:28:49 UTC
(In reply to tmrog from comment #2)
> How could you tell that iOS was killing my extension?  That would be really
> good to know how to do.

There's a crash report called 'JetsamEvent', with a entry whose "reason" field is "per-process-event":

https://gist.github.com/rolfbjarne/bd1bea28fc21508ccf6192526dbece19#file-jetsamevent-2017-03-10-140959-ips-L3596

and the "name" field is your executable:

https://gist.github.com/rolfbjarne/bd1bea28fc21508ccf6192526dbece19#file-jetsamevent-2017-03-10-140959-ips-L3599

In this case, it's because rpages = 640: https://gist.github.com/rolfbjarne/bd1bea28fc21508ccf6192526dbece19#file-jetsamevent-2017-03-10-140959-ips-L3595, and 640 pages * 16kb per page = 10MB.
Comment 4 tmrog 2017-03-13 11:45:54 UTC
Is there a trick to finding these logs?  I don't seem to be able to find them in XCode device logs or on my Mac after a sync.
Comment 5 tmrog 2017-03-13 12:21:42 UTC
I see that corporate policy has "Don't Send" set for Diagnostics and Usage Data.  Does this prevent the log files from being created?
Comment 6 Rolf Bjarne Kvinge [MSFT] 2017-03-14 10:25:31 UTC
It looks like Xcode does not handle this type of crash reports correctly, Xcode processes it somehow, badly, and it ends up corrupted: https://www.dropbox.com/s/2c48yvnbxs8dj2t/Screenshot%202017-03-14%2011.12.34.png?dl=0

However I was able to get these crash reports by using iTunes, just like Dropbox explains here: https://www.dropbox.com/en/help/5065 (sync device with iTunes, look in ~/Library/Logs/CrashReporter/MobileDevice):

$ grep per-process ~/Library/Logs/CrashReporter/MobileDevice/*/JetsamEvent*.ips -B 1 -A 3
    "rpages" : 640,
    "reason" : "per-process-limit",
    "pid" : 1690,
    "idleDelta" : 12892,
    "name" : "FileProvider",
Comment 7 tmrog 2017-03-14 13:31:30 UTC
Do you have Diagnostics and Usage Data set to automatically send to Apple?
Comment 8 Rolf Bjarne Kvinge [MSFT] 2017-03-14 13:34:03 UTC
(In reply to tmrog from comment #7)
> Do you have Diagnostics and Usage Data set to automatically send to Apple?

I just tried disabling it, and the JetsamEvent report is still created.
Comment 9 tmrog 2017-03-14 14:15:35 UTC
Very strange.  Today I am getting them. My rpages is 2560 which is 40MB.  I wonder why the difference?  I am running iOS 10.2.1 on an iPhone6.

  {
    "uuid" : "XXX",
    "states" : [
      "frontmost"
    ],
    "killDelta" : 5534,
    "lifetimeMax" : 5288,
    "age" : 2270786529,
    "purgeable" : 0,
    "fds" : 50,
    "genCount" : 0,
    "coalition" : 10,
    "rpages" : 2560,
    "reason" : "per-process-limit",
    "pid" : 291,
    "idleDelta" : 14320546,
    "name" : "FileProvider",
    "cpuTime" : 0.562036
  },
Comment 10 Rolf Bjarne Kvinge [MSFT] 2017-03-14 15:09:53 UTC
My guess would be that the page size is 4kb instead of 16kb (so that would be 2560 pages * 4kb = 10mb). In the end the memory limit is the same.

One potential reason is that you're running in a 32-bit process instead of 64-bit process.
Comment 11 tmrog 2017-03-15 12:35:07 UTC
You are right the page size 4kb.  Are settings wrong in my FileProvider project or is it dependent on the host app?

BTW...I found this JSON library which doesn't seem to kill the FileProvider.  It doesn't handle things as seamlessly as JSON.net and I have had to modify it for parsing DateTime.  Also, doesn't seem to handle enums so I am trying to figure out how to make that work.

https://github.com/facebook-csharp-sdk/simple-json
Comment 12 Rolf Bjarne Kvinge [MSFT] 2017-03-15 12:37:49 UTC
(In reply to tmrog from comment #11)
> You are right the page size 4kb.  Are settings wrong in my FileProvider
> project or is it dependent on the host app?

A 4kb page size is completely normal (iOS decides it), you only have to be aware if you want to calculate the memory limit from the rpages value in the crash report.
Comment 13 tmrog 2017-03-15 14:04:09 UTC
Rolf,

So, I got SimpleJson working but now have just kicked the problem down the road.  When I try to make a web request using HttpClient and HttpClientHandler my file provider gets terminated.

Questions:

1. It is my understanding the NSUrlSessionHandler uses less memory.  I would love to use that but cannot due to: https://bugzilla.xamarin.com/show_bug.cgi?id=53220.  Any ideas?

2. Any other way to get my memory footprint down?

3. Is it possible to write my FileProvider with Xcode and provide it with my Xamarin app?
Comment 14 Rolf Bjarne Kvinge [MSFT] 2017-03-15 14:10:37 UTC
(In reply to tmrog from comment #13)
> Rolf,
> 
> So, I got SimpleJson working but now have just kicked the problem down the
> road.  When I try to make a web request using HttpClient and
> HttpClientHandler my file provider gets terminated.
> 
> Questions:
> 
> 1. It is my understanding the NSUrlSessionHandler uses less memory.  I would
> love to use that but cannot due to:
> https://bugzilla.xamarin.com/show_bug.cgi?id=53220.  Any ideas?

You can use the iOS API instead of the managed API (NSUrlSession for instance: https://developer.xamarin.com/api/type/Foundation.NSUrlSession/)

> 2. Any other way to get my memory footprint down?
> 
> 3. Is it possible to write my FileProvider with Xcode and provide it with my
> Xamarin app?

Yes, this is possible, but not trivial.

I helped another customer do this here: https://bugzilla.xamarin.com/show_bug.cgi?id=43985#c12, try following those steps, and just ask (in this bug) if you run into problems and I'll help you.
Comment 15 tmrog 2017-03-15 19:20:35 UTC
Rolf,

I finally got my file provider working by doing as you said and using NSUrlSession directly.  That allowed me to make web requests.  The final piece was to use a download task instead of the data task for the file download.  Sounds, of course, obvious but initially just used the same code I had written for making other requests.

Is there a way for me to programmatically determine how much memory I am using?  Hard to monitor it in other ways as the File Provider seems to come and go so quickly.

Also, the only assemblies being loaded now are below.  Are those the minimum assuming I need the Plugin.Settings package?

Loaded assembly: /.monotouch-64/System.dll [External]
Loaded assembly: /.monotouch-64/Xamarin.iOS.dll [External]
Loaded assembly: /.monotouch-64/Mono.Dynamic.Interpreter.dll [External]
Loaded assembly: /.monotouch-64/System.Core.dll [External]
Loaded assembly: /.monotouch-64/Plugin.Settings.Abstractions.dll [External]
Loaded assembly: /.monotouch-64/Plugin.Settings.dll [External]
Loaded assembly: /.monotouch-64/FileProvider.dll

Ted
Comment 16 Rolf Bjarne Kvinge [MSFT] 2017-03-16 06:24:36 UTC
(In reply to tmrog from comment #15)
> 
> Also, the only assemblies being loaded now are below.  Are those the minimum
> assuming I need the Plugin.Settings package?
> 
> Loaded assembly: /.monotouch-64/System.dll [External]
> Loaded assembly: /.monotouch-64/Xamarin.iOS.dll [External]
> Loaded assembly: /.monotouch-64/Mono.Dynamic.Interpreter.dll [External]
> Loaded assembly: /.monotouch-64/System.Core.dll [External]
> Loaded assembly: /.monotouch-64/Plugin.Settings.Abstractions.dll [External]
> Loaded assembly: /.monotouch-64/Plugin.Settings.dll [External]
> Loaded assembly: /.monotouch-64/FileProvider.dll

This looks good; but if curious you can look at each assembly's dependencies by using the monodis tool:

$ monodis --assemblyref /path/to/System.dll
AssemblyRef Table
1: Version=2.0.5.0
	Name=mscorlib
	Flags=0x00000000
	Public Key:
0x00000000: 7C EC 85 D7 BE A7 79 8E 
2: Version=2.0.5.0
	Name=System.Xml
	Flags=0x00000000
	Public Key:
0x00000000: 7C EC 85 D7 BE A7 79 8E 

This shows that System.dll references mscorlib.dll and System.Xml.dll.

Rolf
Comment 17 tmrog 2017-03-16 11:28:10 UTC
Rolf,

Thanks for all your help on this.  I guess in summary for anybody who reads this bug, I had to do the following things to get my file provider extension to work:

1. Remove Json.net from the project.
2. Used simple-json instead - https://github.com/facebook-csharp-sdk/simple-json
3. Stop using managed HTTP layer.  Remove System.Net.Http from the project
4. Used NSUrlSession directly.
5. Moral of the story is, you must use as little memory as possible.  Reduce the assemblies you use in addition to any other memory consumption.

Ted
Comment 18 Rolf Bjarne Kvinge [MSFT] 2017-07-19 07:13:52 UTC
*** Bug 58167 has been marked as a duplicate of this bug. ***
Comment 19 Rolf Bjarne Kvinge [MSFT] 2017-07-19 17:26:44 UTC
*** Bug 58167 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.