Bug 17678 - HttpResponseMessage.Content.ReadAsStringAsync does not remove UTF8 BOM
Summary: HttpResponseMessage.Content.ReadAsStringAsync does not remove UTF8 BOM
Alias: None
Product: Class Libraries
Classification: Mono
Component: System ()
Version: 3.2.x
Hardware: PC Windows
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
Depends on:
Reported: 2014-02-10 04:23 UTC by softlion
Modified: 2016-11-22 11:57 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on GitHub or Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description softlion 2014-02-10 04:23:05 UTC
In a PCL i use the Microsoft.Http nuget to call a json api, which answers JSON in UTF8 (using the standard .NET 4.5.1 JavaScript serializer). Then i use JSON.NET to parse the result as json.
JSON.NET throws that json is invalid, as it does not start with a "{"


    using (var wc = new HttpClient())
       response = await wc.SendAsync(message, CancellationToken.None).ConfigureAwait(false);

    stringResult = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
    result = JObject.Parse(stringResult);

When examining stringResult while debugging the Android app, it starts with the 3 bytes BOM !!
For BOM see http://en.wikipedia.org/wiki/Byte_order_mark

This seems to be a bug in the mono implementation of ReadAsStringAsync, as the bug does not appear in the unit test which uses MS unit test project (and legacy .NET implementation).

Xamarin.Android version 4.10.02014
Comment 1 softlion 2014-02-10 04:35:34 UTC
After more tests, only the first char is bad as it is 0xFF. Second char is start of Json "{". So nothing to do with BOM.
Comment 2 Miguel de Icaza [MSFT] 2014-02-11 21:40:38 UTC
Could you provide the before/after contents of the stringResult in both Windows and Android?

Please dump the string contents with something like:

foreach (var c in stringResult)
    Console.Write ("{0:x2}", (int) c)
Comment 3 softlion 2014-02-12 00:36:04 UTC
Image of what you want:

Sender code: (asp.net mvc):
        public ActionResult GenerateNiceIds(int count, NiceIdItemType type, int extra)
            if(count <=0 || count >= 5000)
                throw new ArgumentOutOfRangeException("count", "This should never happen");

            var niceIdManager = Mvx.Resolve<INiceIdManager>();
            var niceIds = new List<string>(count);
                niceIds.Add(niceIdManager.NewId(type, extra));

            return Json(new { ok=true, niceIds });

Receiver code extract:

                    response = await wc.SendAsync(message, CancellationToken.None).ConfigureAwait(false);

                if (response.Content != null)
                    stringResult = await response.Content.ReadAsStringAsync().ConfigureAwait(false);

                    if (response.Content.Headers != null && response.Content.Headers.ContentType != null
                        && response.Content.Headers.ContentType.MediaType == "application/json")
                        //bug in Android version of ReadAsStringAsync (reported): temporary patch. Extra 0xFF at start ...
                        if (stringResult.Length >= 1 && (byte)stringResult[0] == 0xFF)
                            stringResult = stringResult.Substring(1);
Comment 4 Miguel de Icaza [MSFT] 2014-02-12 09:43:47 UTC
Mhm, not sure this helped me very much.

Would you mind hosting a web site that exposes the problem and give me the client code that reproduces the bug?
Comment 5 softlion 2014-02-25 08:52:27 UTC
Well, it works ok in one project, not at all on another which need the fix.
Both are using the same code calling a ws and getting the json result.

I'll try to create a demo.
Comment 6 softlion 2014-07-29 14:47:29 UTC
Still buggy in current stable ios release, with PCL profile Windows 8+NET 4.5+Android+iOS.
The UTF8 BOM is still not trimmed.

A workaround is to switch to profile Silverlight8+NET45+Android+iOS and install the HTTPClient PCL in nuget.
Comment 7 Marek Safar 2016-04-16 09:36:20 UTC
Is this issue still happening for you with Mono 4.4?
Comment 8 softlion 2016-04-16 19:14:09 UTC
I don't know, since this problem i only use the native HttpClient on ios or android by registering it in an iOS containter.

This is also the only way to get the proxy set on ios to work (httpclient does bypass it happily), and also to activate gzip transmission.

Btw if i use the new option "replace httpclient by NSUrlSession" i have a problem when i use httpclient as a singleton (ie: i use the same httpclient object in different app services that may run simultaneously). There is a CancellationToken is already disposed exception.
Comment 9 softlion 2016-04-16 19:14:37 UTC
iOC container, not iOS ... So it is available in PCL libs.
Comment 10 softlion 2016-11-22 11:57:05 UTC
Seems fixed now, but i always use the platform version through IOC instead of the PCL version directly now on.