Bug 29928

Summary: Problems with encodings
Product: iOS Reporter: Sebastian <buero>
Component: XI runtimeAssignee: Sebastien Pouliot <sebastien>
Status: VERIFIED FIXED    
Severity: normal CC: atsushi, gouri.kumari, jonp, kumpera, masafa, miguel, mono-bugs+monotouch, rolf, sebastien, udhams
Priority: Normal    
Version: XI 8.10   
Target Milestone: 8.13 (C6 alpha)   
Hardware: Macintosh   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:

Description Sebastian 2015-05-11 06:02:55 UTC
The current version of Xamarin.iOS 8.10 seems to have a problem with encodings. In two of my apps string results from an api call get directly displayed on an UILabel show question marks instead of the proper umlaut. App versions that get build with Xamarin.iOS 8.9.1.3 display strings like "Müller" correctly while apps build with 8.10 does not. Strings displayed from a lproj localizable.strings file do not show this effect.

Steps to reproduce on Mac:

1) Install Xamarin.iOS 8.9.1.3
2) Make a web request with a response that contains some umlauts like ä, ü or ß using WebClient
3) Display the result
4) Repeat with Xamarin.iOS 8.10


=== Xamarin Studio ===

Version 5.9 (build 431)
Installation UUID: 7e36f40e-122f-4090-be39-a0ab2eed880e
Runtime:
	Mono 4.0.0 ((detached/d136b79)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 400000143

=== Xamarin.Android ===

Version: 5.1.0.115 (Business Edition)
Android SDK: /Users/Sebastian/Library/Developer/Xamarin/android-sdk-mac_x86
	Supported Android versions:
		4.0.3  (API level 15)
		4.2    (API level 17)
		4.3    (API level 18)
		4.4    (API level 19)
		4.4.87 (API level 20)
		5.0    (API level 21)
Java SDK: /usr
java version "1.7.0_71"
Java(TM) SE Runtime Environment (build 1.7.0_71-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.71-b01, mixed mode)

=== Xamarin Android Player ===

Not Installed

=== Apple Developer Tools ===

Xcode 6.2 (6776)
Build 6C131e

=== Xamarin.iOS ===

Version: 8.9.1.3 (Business Edition)
Hash: f7736a4
Branch: 
Build date: 2015-04-09 04:22:08-0400

=== Xamarin.Mac ===

Not Installed

=== Build Information ===

Release ID: 509000431
Git revision: 7560726734fc7267de2fa9abed2509968deefaa8
Build date: 2015-04-17 19:25:48-04
Xamarin addins: 2e772c734ab3148054eae7bf8949f340fdeb5e5e

=== Operating System ===

Mac OS X 10.10.3
Darwin Sebastians-iMac.local 14.3.0 Darwin Kernel Version 14.3.0
    Mon Mar 23 11:59:05 PDT 2015
    root:xnu-2782.20.48~5/RELEASE_X86_64 x86_64
Comment 1 Rolf Bjarne Kvinge [MSFT] 2015-05-11 07:31:19 UTC
This sounds like there's an encoding problem between your app and the web service.

Can you attach a self-contained test project we can use to reproduce this?
Comment 2 Sebastian 2015-05-11 07:55:15 UTC
At first I thought that too. Building the same app from same source with different Xamarin.iOS versions proofs that it is not the case. I also see this behavior at different apps with different webservices.

Let me see if I can find a public available web service with some umlauts to build a test project.
Comment 4 Rolf Bjarne Kvinge [MSFT] 2015-05-12 04:49:32 UTC
This is because we ask iOS what's the default encoding is now.

Previously we hardcoded the default encoding to be UTF8: 

https://github.com/mono/mono/blob/mono-3.12.0-branch/mcs/class/corlib/System.Text/Encoding.cs#L855

> defaultEncoding = UTF8;

and now we ask the OS:

https://github.com/mono/mono/blob/master/mcs/class/corlib/System.Text/EncodingHelper.cs#L81 
> string code_page_name = InternalCodePage (ref code_page);

which ends up here:

https://github.com/mono/mono/blob/master/eglib/src/gunicode.c#L225

> my_charset = nl_langinfo (CODESET);

The workaround is simple, just set the ending for the web client:

> webclient.Encoding = Encoding.UTF8;
Comment 5 Rolf Bjarne Kvinge [MSFT] 2015-05-12 04:50:39 UTC
@Sebastien, should we keep the new behavior (ask the OS for the default encoding), or should be revert to the old one (hardcode UTF8 as the default encoding)?
Comment 6 Sebastien Pouliot 2015-05-12 08:45:46 UTC
I'd rather keep (or switch back to) the old behaviour, otherwise this issue will keep coming bite back for a long time.

OTOH that sounds like a breaking change in Mono 4.0 that would affect other products as well. I don't recall this being mentioned anywhere (including the release notes [1]). C.c. relevant parties

[1] http://www.mono-project.com/docs/about-mono/releases/4.0.0/
Comment 7 Marek Safar 2015-05-12 08:57:23 UTC
You won't see any note in mono release notes because mono behaviour didn't change. The hack was there for mobile only. I don't know why eno didn't keep this behaviour but it makes  sense not to have breaking change for mobile here.
Comment 8 Sebastien Pouliot 2015-05-12 15:01:38 UTC
@Atsushi any specific reason (for XA or general) for the change ? 

Otherwise I'd like to go back to the old (UTF8) default (minimally for XI, likely XM too).
Comment 9 Jonathan Pryor 2015-05-12 17:13:06 UTC
@Spouliot: For one thing, the old UTF-8 was broken; see Bug #23771, which in turn appears to be responsible for Bug #14918 (Xamarin.Android).

Would it be possible to override/set the default encoding during process startup?
Comment 10 Atsushi Eno 2015-05-12 21:12:02 UTC
Define "change"?

What I "changed" was to switch to referencesource, nothing else.
Comment 11 Sebastien Pouliot 2015-05-13 08:39:04 UTC
@Jon I'm not sure that bugs in the (old) UTF8 encoder are a good reasons to switch our default encoding (even less since the new encoder should not have those issues). If you think the current code is better suited for XA then we'll change this for a XI/XM specific define (it's not harder for us).

@Atsushi "change" as pointed out in comment #4, i.e. removal of `#if MOBILE` to get the default encoding. That code was copied from Encoding.cs to EncodingHelper.cs without that part.
Comment 12 Sebastien Pouliot 2015-05-22 09:56:37 UTC
mono/master 605192528de1a9c9c95a97bc5a2de7d3c85b322d restore MONOTOUCH/XAMMAC to use the UTF8 default (as MOBILE used to)

I'll see to get this backported into 8.10 SR2.
Comment 13 Sebastien Pouliot 2015-05-25 16:10:42 UTC
maccore/master 88bd63225b417e3618cc804fe8d5533586953c3b bumped mono, this can now be verified
Comment 17 Sebastien Pouliot 2015-05-26 15:22:27 UTC
@Udham this change causing a regression in some tests. I'l give you a newer revision to verify asap.
Comment 19 Sebastien Pouliot 2015-05-27 13:42:59 UTC
The original fix (using UTF8Unmarked) uncovered issues in the System and SystemNet.Http unit tests. It also did not match exactly the old UTF8 settings. For those reason is was reverted.

maccore/master 99a1e140861b6585b9d63ceed79d77a37cf13b2b has a different fix, which use UTF8 with the same configuration as older (XI 8.9 and previous) releases.
Comment 20 Udham Singh 2015-05-28 14:09:58 UTC
We have checked this issue with X.iOS 8.11.0.880 and followed the steps mentioned in comment 18, and observed that we are not getting any machine in response with X.iOS 8.11.0.880. In application output we are getting "utf-8" but on iOS device screen we are not getting any machine in response.

iOS device iPhone 5 (iOS 7.1) / iPhone 4S (iOS 8.4)  screen : http://www.screencast.com/t/8M7VyH63zCVa

Steps I followed : 

1. Set iOS device iPhone 5 (iOS 7.1) to German-Germany(International > Region Format)
2. Open the demo project provided in comment 3.
3. Implement below code in method 'FinishedLaunching'.
> Console.WriteLine (System.Text.Encoding.Default.WebName);
4. Run the app on iOS device and get below in application output
> "utf-8"

Screencast for application Output : http://www.screencast.com/t/d8xt7Xzn4K9

iOS Device Log : https://gist.github.com/Udham1/0454634b577e0d779529
Application Output : https://gist.github.com/Udham1/6c4c8be06820e6e05981
Build output : https://gist.github.com/Udham1/1f750b19bd4ce9df300a
Ide Log : https://gist.github.com/Udham1/41b82f4623392466cce6

Note : 
1. This is working fine with X.iOS 8.11.0.857 as mentioned in comment 18.
2. However with X.iOS 8.10.1.64 and X.iOS 8.10.0.303, we are getting "us-ascii" in application output and on iOS device we are getting first machine as "M??nchoff-Rimpau GbR"

Environment Info :

=== Xamarin Studio ===

Version 5.9.2 (build 2)
Installation UUID: ce927b2a-2c07-44c5-b186-09cfdafba6dc
Runtime:
	Mono 4.0.1 ((detached/ed1d3ec)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 400010044

=== Apple Developer Tools ===

Xcode 6.2 (6776)
Build 6C131e

=== Xamarin.iOS ===

Version: 8.11.0.880 (Enterprise Edition)
Hash: 99a1e14
Branch: master
Build date: 2015-05-28 07:56:51-0400

=== Build Information ===

Release ID: 509020002
Git revision: 43735a4b905b571f69cf9428a3ea7b52aef55476
Build date: 2015-05-18 09:38:12-04
Xamarin addins: ab18ddff45f507ed74f36c2b65df9aee22e28a56

=== Operating System ===

Mac OS X 10.9.5
Darwin Xamarin76s-Mac-mini.local 13.4.0 Darwin Kernel Version 13.4.0
    Sun Aug 17 19:50:11 PDT 2014
    root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64
Comment 22 Sebastien Pouliot 2015-05-28 14:32:10 UTC
@Udham the list comes form a server (web service). If it's empty it's likely because the service was done (or the list it returned was empty).

> 2015-05-28 23:06:48.972 EncodingDemo2[319:60b] utf-8

^ The fact that it returns "uff-8" shows that the code was changed (and back to it's old default).

Just to be sure can you retry this ? but let's wait for @Sebastian to give you to go ahead.

Also just use in case the above revision had a different issue (with networking or web services) please try the latest build available from master
Comment 23 Sebastian 2015-05-28 14:46:25 UTC
@Udham @Sebastien Service should be up and running again. Tested it with the provided demo project. Can you confirm?
Comment 24 Sebastien Pouliot 2015-05-28 14:55:55 UTC
Yes, I get the list just fine.

@Udham please try again
Comment 25 Udham Singh 2015-05-28 15:09:08 UTC
I have checked this again but still not getting the list. I have checked this with 2 devices iPhone 5 (iOS 7.1) / iPhone 4S (iOS 8.4) and not getting the list with both the devices.

Screencast : http://www.screencast.com/t/2vRJQoWvSh

Environment info : Same build as mentioned in comment 20.
Comment 26 Sebastien Pouliot 2015-06-01 14:44:22 UTC
I could not reproduce myself but Gouri had some random issues with the web service on Friday.

> The values didn’t appear during the first few attempts. But now it is appearing consistently,

However the printed text (utf-8) means it's unrelated to the fix itself - and this should not stop us from backporting to SR2.
Comment 28 Udham Singh 2015-06-03 10:47:03 UTC
I have checked this issue with X.iOS 8.10.2.0 (latest build from C5SR2) and iPhone 5 (iOS 7.1). I am getting the list on device and in application output "utf-8", Hence I am closing this issue.

To verify this issue I have followed the same steps mentioned in comment 18.

Screencast for application Output : http://www.screencast.com/t/oYabztNO
Screencast for iOS device screen : http://www.screencast.com/t/npNG76uUmG3f

Environment Info :

=== Xamarin Studio ===

Version 5.9.2 (build 4)
Installation UUID: ce927b2a-2c07-44c5-b186-09cfdafba6dc
Runtime:
	Mono 4.0.1 ((detached/ed1d3ec)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 400010044

=== Apple Developer Tools ===

Xcode 6.2 (6776)
Build 6C131e

=== Xamarin.iOS ===

Version: 8.10.2.0 (Trial Edition)
Hash: 6403a1e
Branch: master
Build date: 2015-06-02 23:12:27-0400

=== Xamarin.Android ===

Version: 5.1.2.1 (Trial Edition)
Android SDK: /Users/xamarin76/Desktop/android-sdk-macosx
	Supported Android versions:
		2.3    (API level 10)
		4.0.3  (API level 15)
		4.1    (API level 16)
		4.2    (API level 17)
		4.3    (API level 18)
		4.4    (API level 19)
		4.4.87 (API level 20)
		5.0    (API level 21)
Java SDK: /usr
java version "1.7.0_65"
Java(TM) SE Runtime Environment (build 1.7.0_65-b17)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

=== Xamarin Android Player ===

Version: Unknown version
Location: /Applications/Xamarin Android Player.app

=== Xamarin.Mac ===

Version: 2.0.1.64 (Starter Edition)

=== Build Information ===

Release ID: 509020004
Git revision: 288eb01e990f2d93b856c9600a0c40708b930128
Build date: 2015-06-01 14:29:52-04
Xamarin addins: c12ed369b0e05dc5c35c9350fdcbc00a46c0ff99

=== Operating System ===

Mac OS X 10.9.5
Darwin Xamarin76s-Mac-mini.local 13.4.0 Darwin Kernel Version 13.4.0
    Sun Aug 17 19:50:11 PDT 2014
    root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64