Bug 5827 - Deserializing xml with strange ASCII characters throws an exception
Summary: Deserializing xml with strange ASCII characters throws an exception
Alias: None
Product: iOS
Classification: Xamarin
Component: Xamarin.iOS.dll ()
Version: 5.3.x
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: 6.4 (async)
Assignee: Bugzilla
: 11088 ()
Depends on:
Reported: 2012-06-22 14:28 UTC by Blake Jones
Modified: 2013-07-15 13:12 UTC (History)
10 users (show)

Is this bug a regression?: ---
Last known good build:

Sample solution that shows serialization issue (7.26 KB, application/zip)
2013-03-12 20:16 UTC, Andy
Updated solution to encode bad characters, issue now matches description (7.59 KB, application/zip)
2013-03-12 21:47 UTC, Andy
Updated solution, Setting CheckCharacters in XmlReaderSettings (7.73 KB, application/zip)
2013-03-13 01:14 UTC, Andy
Includes WCF solution to fully duplicate issue (199.65 KB, application/zip)
2013-03-13 09:29 UTC, Andy

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Blake Jones 2012-06-22 14:28:46 UTC
I have a WCF service which returns text from files. However some of the files that I deal with contain strange ASCII values and when they are deserialized on the client the deserializer throws an exception, which I've attached to the end of this description.

I ran a test and the ASCII characters that throw an exception are those with a decimal value from 0 to 8, 11, 12, and 14 to 31.

On a normal WCF client those characters do not throw an exception.

The following exception is for a string that only contains the ASCII character with a decimal value of 0.

System.Xml.XmlException: Referenced character was not allowed in XML. Normalization is True, checkCharacters = True  Line 1, position 170.
  at Mono.Xml2.XmlTextReader.ReadCharacterReference () [0x00165] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1775
  at Mono.Xml2.XmlTextReader.ReadReference (Boolean ignoreEntityReferences) [0x00015] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1728
  at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00048] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1663
  at Mono.Xml2.XmlTextReader.ReadContent () [0x0015c] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1359
  at Mono.Xml2.XmlTextReader.Read () [0x00141] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader.cs:626
  at System.Xml.XmlTextReader.Read () [0x0006b] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlTextReader2.cs:564
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Xml/XmlSimpleDictionaryReader.cs:253
  at System.Xml.XmlReader.ReadStartElement () [0x0002f] in /Developer/MonoTouch/Source/mono/mcs/class/System.XML/System.Xml/XmlReader.cs:724
  at System.Xml.XmlDictionaryReader.ReadElementContentAsString () [0x00018] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Xml/XmlDictionaryReader.cs:397
  at System.Runtime.Serialization.XmlFormatterDeserializer.DeserializePrimitive (System.Type type, System.Xml.XmlReader reader, System.Xml.XmlQualifiedName qname) [0x000c1] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Runtime.Serialization/XmlFormatterDeserializer.cs:229
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Type type, System.Xml.XmlReader reader) [0x0028e] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Runtime.Serialization/XmlFormatterDeserializer.cs:189
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Xml.XmlReader reader, System.Type declaredType, System.Runtime.Serialization.KnownTypeCollection knownTypes, IDataContractSurrogate surrogate, System.Runtime.Serialization.DataContractResolver resolver, System.Runtime.Serialization.DataContractResolver defaultResolver, System.String name, System.String ns, Boolean verifyObjectName) [0x0007e] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Runtime.Serialization/XmlFormatterDeserializer.cs:65
  at System.Runtime.Serialization.DataContractSerializer.ReadObject (System.Xml.XmlDictionaryReader reader, Boolean verifyObjectName) [0x00024] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Runtime.Serialization/DataContractSerializer.cs:344
  at System.Runtime.Serialization.XmlObjectSerializer.ReadObject (System.Xml.XmlDictionaryReader reader) [0x00000] in /Developer/MonoTouch/Source/mono/mcs/class/System.Runtime.Serialization/System.Runtime.Serialization/XmlObjectSerializer.cs:74
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.ReadMessagePart (System.ServiceModel.Description.MessagePartDescription part, System.Xml.XmlDictionaryReader r) [0x00032] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseMessagesFormatter.cs:439
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.MessageToParts (System.ServiceModel.Description.MessageDescription md, System.ServiceModel.Channels.Message message) [0x000ce] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseMessagesFormatter.cs:418
  at System.ServiceModel.Dispatcher.BaseMessagesFormatter.DeserializeReply (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x0004d] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseMessagesFormatter.cs:273
  at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeReply (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x00000] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseMessagesFormatter.cs:90
  at System.ServiceModel.MonoInternal.ClientRuntimeChannel.Request (System.ServiceModel.Description.OperationDescription od, System.Object[] parameters) [0x0024d] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel/ClientRuntimeChannel.cs:570
  at System.ServiceModel.MonoInternal.ClientRuntimeChannel.DoProcess (System.Reflection.MethodBase method, System.String operationName, System.Object[] parameters) [0x00038] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel/ClientRuntimeChannel.cs:502
  at System.ServiceModel.MonoInternal.ClientRuntimeChannel.Process (System.Reflection.MethodBase method, System.String operationName, System.Object[] parameters) [0x0001c] in /Developer/MonoTouch/Source/mono/mcs/class/System.ServiceModel/System.ServiceModel/ClientRuntimeChannel.cs:479
Comment 1 Rolf Bjarne Kvinge [MSFT] 2013-03-12 18:53:34 UTC
Can you create a test project we can use to reproduce this? Otherwise it will be impossible to track down.
Comment 2 Rolf Bjarne Kvinge [MSFT] 2013-03-12 18:54:06 UTC
*** Bug 11088 has been marked as a duplicate of this bug. ***
Comment 3 Andy 2013-03-12 20:16:32 UTC
Created attachment 3587 [details]
Sample solution that shows serialization issue

This sample only uses the DataContractSerializer, which renders a slighlty different message than when used in WCF, but keeping the example as atomic and simple as possible.
Comment 4 Andy 2013-03-12 21:47:10 UTC
Created attachment 3588 [details]
Updated solution to encode bad characters, issue now matches description

The previous solution generated an exception but not the exact same one because the bad characters weren't encoded. This solution encodes the bad characters and results in the exact error message.
Comment 5 Andy 2013-03-13 01:14:38 UTC
Created attachment 3589 [details]
Updated solution, Setting CheckCharacters in XmlReaderSettings

Sorry to keep posting updated solutions, but trying to help as I discover more.  Looking through the trace and at the actual code, I believe the issue is the CheckCharacters.  If this is explicitly set to false, the serialization succeeds.  Unfortunately I don't see a way to configure WCF's xml settings for the deserialization. Not sure if this should be unset for the WCF code.
Comment 6 Rolf Bjarne Kvinge [MSFT] 2013-03-13 07:05:26 UTC
Both sample solutions in comment 3 and 4 throws the exception on .NET too: "There was an error deserializing the object of type SerializationIssue.TestClass. '♠', hexadecimal value 0x06, is an invalid character. Line 1, position 186. " - what I'd need is a sample that works on .NET but fails in Mono (it doesn't necessarily have to be a small sample, as long as it's easy to trigger the problem).
Comment 7 Andy 2013-03-13 09:29:33 UTC
Created attachment 3598 [details]
Includes WCF solution to fully duplicate issue

This contains a WCF Service, a Silverlight client that correctly serializes the bad characters and a Mono client that does not. It will have to be configured appropriately for your environment.
Comment 8 Rolf Bjarne Kvinge [MSFT] 2013-03-13 17:31:50 UTC
I was not able to open the TestWcfService project in Visual Studio 2010 (This project is incompatible with the current version of Visual Studio) - is there anything extra I need to install for VS to recognize it?
Comment 9 Andy 2013-03-13 18:00:06 UTC
I'm using VS2012, I don't have VS2010 currently installed.  I'm not sure what is needed to get it working in 2010.
Comment 11 Andy 2013-03-13 18:05:30 UTC
target framework is also 4.5.  Not sure if 2010 supports that still?
Comment 12 Andy 2013-03-13 22:42:52 UTC
I was also thinking that the TestWCFService project is just a basic WCF Service project. you could generate one from scratch and copy the files over.  You would just have to configure it to host the Silverlight application as well to test the Silverlight app.
Comment 15 Atsushi Eno 2013-03-19 08:01:06 UTC
Character references to those character ranges indeed violates XML 1.0 specification. Hence this is *the* exact expected behavior. Any valid XML parsers should reject as a not-well-formed XML document.

Any SOAP web services that contain such XML responses that contain invalid characters need to get fixed, not the client. You had better check first if .NET processes such invalid XML responses with this default configuration.

As far as I knew from months ago, mono WCF does not fully support configurations (especially very detail part of them). So the same configuration on desktop may not work. Anyways in mobile configuration is not supported at all, so if the difference comes from configuration, they have to be represented in code. There might be some configuration settings that allows it and turns on a switch to set "XmlReaderSettings.CheckCharacters = false" in the client's message consumer.
Comment 17 Atsushi Eno 2013-03-19 09:46:33 UTC
Silverlight and WCF are different in implementation (e.g. Silverlight allows some invalid JSON either) so it could still be that Silverlight *incorrectly* handles this situation (as I explained earlier, it is incorrect to NOT reject this invalid XML).

Anyhow, it should be also noted that it is also possible that the "default" configuration for desktop is not fully "implemented". WCF as of .NET 4.0 introduced the "simplified" configuration settings that brings a lot of default settings that may have changed the behavior of WCF as of .NET 3.5. In that case it is not Message implementation but something in System.ServiceModel.Configuration.
Comment 18 Andy 2013-03-19 10:24:56 UTC
I understand that a valid XML 1.0 parser should reject invalid XML.  .NET's implementation of WCF (both in 3.5 and 4.0) appear to allow this without any configuration options (that I could find) to handle invalid XML characters.  I don't know why, but perhaps they do this because they do not want to put limitations on the data that can be sent over the wire.  Maybe that's why they created the DataContractSerializer so they could xml serialize any data even if the result didn't conform to the specification?  This could also be the reason why the Silverlight client has the same incorrect behavior.  The result of both sides (WCF and Silverlight) handling the XML incorrectly is that any data (even if its invalid XML data) can be transmitted.  Perhaps .NET should have implemented a double encoding solution or maybe should have implement a later version of the XML spec that can handle invalid characters.

All that being said, since Mono is attempting to emulate Silverlight and use the proxy generated from the tool, shouldn't it match the incorrect behavior and process the bad XML?

As an aside, I'm not sure how relevant this is but when using XML-web services and the .net proxy it generates, the service does handle the invalid XML without exception.  I didn't dig deep, so I don't know why for sure, but might be helpful or relevant.
Comment 19 Atsushi Eno 2013-03-19 11:21:19 UTC
Yeah maybe those people behind mono WCF will "fix" the issue. (I never support infecting mono with bad XML implementation by myself. I gave hint at comment #15 which should be enough.)
Comment 20 Bryan Moulton 2013-03-19 17:16:28 UTC
Consider the following example for creating a file reader that automatically removes unacceptable characters. Implement something like this in your server code.
Comment 21 Martin Baulig 2013-03-21 14:35:10 UTC
Would the following qualify as valid XML ?

HTTP/1.1 200 OK
Cache-Control: private
Content-Length: 187
Content-Type: text/xml; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Thu, 21 Mar 2013 18:31:08 GMT
X-Charles-Received-Continue: HTTP/1.1 100 Continue

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/"><s:Body><TestResponse xmlns="http://tempuri.org/"><TestResult>Hello &#x6;</TestResult></TestResponse></s:Body></s:Envelope>

Works fine with .NET, but doesn't with Mono.
Comment 23 Martin Baulig 2013-03-21 15:07:01 UTC
Just tested this on .NET:

			var Data = "<TestResult>Hello &#x6;</TestResult>";
			using (var reader = new StringReader(Data))
				var settings = new XmlReaderSettings ();
				settings.CheckCharacters = true;
				var xml = XmlTextReader.Create(reader, settings);
				var content = xml.ReadString();
				Console.WriteLine("TEST: {0:x}", (int)content[6]);

Throws an exception when 'settings.CheckCharacters = true'.

So I think we should just set that to false when using XmlTextReader to deserialize a SOAP response.
Comment 24 Martin Baulig 2013-03-21 16:11:49 UTC
Did some more research on this and XmlObjectSerializer does not check for these invalid characters when you use any of the 'Stream' overloads.  I'll have a fix shortly.

See also http://stackoverflow.com/questions/8215657/how-to-stop-net-xml-serialisation-inserting-illegal-characters.
Comment 25 Martin Baulig 2013-03-21 20:18:41 UTC
mono/master commit 91f11b2 has a partial fix for this, which makes XmlObjectSerializer ignore invalid characters as it's supposed to do.

Unfortunately, this doesn't fix the problem yet.
Comment 26 Martin Baulig 2013-03-21 20:58:29 UTC
mono/master commit d2d903f should now fix it :-)
Comment 28 PJ 2013-07-15 13:12:04 UTC
Setting to RESOLVED FIXED based on comment 26.