Bug 14918

Summary: mandroid daemon not supporting reading accented characters
Product: Android Reporter: André <a.leblanc>
Component: MSBuildAssignee: Jonathan Pryor <jonp>
Status: VERIFIED FIXED    
Severity: major CC: abhishekk, adnvilla, andre.schreiter, antonio_serpa, brendan.zagaeski, cedric, cmarrades, davis.andre, eloekset, estarles.ramille, fred, g-ondaro, ivanfdezf50, jeremie.laval, jmostajo, jonp, j.petereit, jvlppm, leo.locurcio, marcfernandezveiga, masi.c64, mono-bugs+monodevelop, mono-bugs+monodroid, mukesh.mishra, naqeeba, peter.collins, pierre, ralvaradot, robert.fuszenecker, rubencm, tarcisiocms, thiago.sabin, thordur.h, vmprima
Priority: Normal    
Version: 4.20.0   
Target Milestone: 7.0 (C8)   
Hardware: PC   
OS: Windows   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: Test Case

Description André 2013-09-22 22:53:20 UTC
When trying Xamarin and compiling Tasky I got this error:

C:\Program Files (x86)\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(3,3): Error XA0000: Unexpected error - Please fill a bug report at http://bugzilla.xamarin.com. 

Reason: Could not find SDK directory 'C:\Users\André\AppData\Local\Android\android-sdk'.  Is --sdk-dir set appropriately? (XA0000) (TaskyAndroid)


C:\Users\André\AppData\Local\Android\android-sdk

should be read

C:\Users\André\AppData\Local\Android\android-sdk
Comment 1 Jonathan Pryor 2013-09-24 10:36:29 UTC
What locale are you using?
Comment 2 Mikayla Hutchinson [MSFT] 2013-09-24 14:34:52 UTC
Looks like UTF-8 being interpreted as LATIN1.

Does Tools->Options->Projects->SDK Locations->Android show the correct value?
Comment 3 Mikayla Hutchinson [MSFT] 2013-09-24 14:42:59 UTC
I suspect this may have something to do with the chain installer writing incorrectly encoded registry key values.
Comment 4 André 2013-09-25 07:56:12 UTC
I'm on Windows 8 Pro. My folder was create using FR-ca local.
Comment 5 André 2013-09-25 07:57:07 UTC
Yes it was showing OK
Comment 6 André 2013-09-25 07:57:51 UTC
Does Tools->Options->Projects->SDK Locations->Android show the correct value?

Was showing correct values..
Comment 7 Mikayla Hutchinson [MSFT] 2014-02-09 13:40:46 UTC
*** Bug 17667 has been marked as a duplicate of this bug. ***
Comment 8 Mikayla Hutchinson [MSFT] 2014-02-09 13:41:46 UTC
*** Bug 17127 has been marked as a duplicate of this bug. ***
Comment 9 Mikayla Hutchinson [MSFT] 2014-02-09 13:41:55 UTC
*** Bug 17125 has been marked as a duplicate of this bug. ***
Comment 10 Ramille 2014-02-09 16:26:51 UTC
I'm sorry, but I went to check Tools->Options->Projects->SDK Locations->Android, and it is showing the correct value but (just like everybody else, I think) it still doesn't work...

You talked about the chain installer writing incorrectly encoded registry key values. Is there a way to install it correctly ?

What am I suppose to do to make it work, if it can be done ?
Comment 11 Mikayla Hutchinson [MSFT] 2014-02-10 00:40:18 UTC
If the SDK location settings shows the correct name then the issue is probably in our MSBuild code.
Comment 12 Ramille 2014-02-10 13:18:19 UTC
And, can I do something about it ? or am I supposed to download a new version somewhere, somehow ? 

Or is it a problem that will be fixed with a new version witch will come out in... sometime?
Comment 13 Mikayla Hutchinson [MSFT] 2014-02-10 13:50:45 UTC
*** Bug 17688 has been marked as a duplicate of this bug. ***
Comment 14 Jonathan Pryor 2014-02-14 09:51:21 UTC
*** Bug 17776 has been marked as a duplicate of this bug. ***
Comment 15 Thiago 2014-02-14 12:52:24 UTC
I just wanna say that it's not a sdk problem cuz i switched my path to another one withou the character "é" and it worked for a time now it came with this error

C:\Program Files\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(2,2): Error XA0000: Unexpected error - Please file a bug report at http://bugzilla.xamarin.com. Reason: System.UnauthorizedAccessException: Access to the path "C:\Users\Cacupé" is denied. (XA0000) (novoTeste)

Please make this works i need to work
Comment 16 Jonathan Pryor 2014-02-14 17:03:41 UTC
@Thiago: Can you provide Diagnostic build output? Why is it trying to access "C:\Users\Cacupé"? (For `mandroid --sdk-dir`?).

Assuming it's for the Android SDK location, you could try moving it into a directory that only has ASCII characters, then update the path in Tools > Options > Projects > SDK Locations >Android.
Comment 17 Eivind Gussiås Løkseth 2014-02-26 15:26:13 UTC
I'm having the same problem, and I've moved the SDK and NDK to a new path, but still I get this compile error in Visual Studio. It must be a setting somewhere that can be modified. The SDK and NDK paths are set correctly, and so is the ANDROID_NDK_PATH environment variable.

What setting do we need to change?
Comment 18 dean.ellis 2014-03-06 04:04:53 UTC
*** Bug 18203 has been marked as a duplicate of this bug. ***
Comment 19 Prashant manu 2014-05-09 05:01:34 UTC
*** Bug 19634 has been marked as a duplicate of this bug. ***
Comment 20 Mikayla Hutchinson [MSFT] 2014-05-12 18:10:45 UTC
It seems it's not just searching for SDKs that's affected by non-ASCII paths:
http://forums.xamarin.com/discussion/16645/problem-after-installation#latest
Comment 21 Prashant manu 2014-06-02 07:57:57 UTC
*** Bug 20138 has been marked as a duplicate of this bug. ***
Comment 22 Prashant manu 2014-06-03 07:54:38 UTC
*** Bug 20296 has been marked as a duplicate of this bug. ***
Comment 23 Cedric 2014-06-07 09:22:08 UTC
Not sure if this is useful but additional information:

- My locale is fr-fr

- Originally Windows 8 Pro 64, I can't remember but I think the setup was in en-GB 
then changed to fr-fr.

- I see the path correctly in Tools->Options->Projects->SDK Locations->Android

- I just redownloaded the Tshirt App store.
I get informed that it has Unix end of lines & would I like to change to Windows?

I answer yes and get the error message:
Can't save file - access denied
L'accès au chemin d'accès 'C:\Program Files (x86)\MSBuild\Xamarin\Android\.#Xamarin.Android.Common.targets' est refusé.

- When trying to debug the solution I click 'create emulator' and all are in errors. One example:

AVD Détails
Name: MonoForAndroid_API_15
CPU/ABI: ARM (armabi)
Path C:\Users\C?dric\.android\avd\MonoForAndroid API 15.avd
Error: Failed to parse properties from C:\Users\C?dric\.android\avd\MonoForAndroid API 15.avd\config.ini

Here also, C?dric should be Cédric
Comment 24 Cedric 2014-06-07 09:52:48 UTC
Not sure whether I should file another bug report or not.

This time I downloaded the Analog Clock sample.

I opened the solution in Visual Studio 2013 Premium Update 2.

The project build went fine.

When trying to debug the iOS version with the iOS Emulator:

http://hpics.li/404daa6 (screenshot)

This seems correlated as this is a wrong path.

Interestingly, the é From Cédric is replaced by 2 characters.
Comment 25 Rajneesh Kumar 2014-08-08 03:54:10 UTC
*** Bug 21871 has been marked as a duplicate of this bug. ***
Comment 26 Adrián 2014-08-13 18:21:04 UTC
same issue with spanish: Adrián -> Adrián

C:\Program Files (x86)\MSBuild\Xamarin\Android\Xamarin.Android.Common.targets(3,3): Error XA0000: Unexpected error - Please file a bug report at http://bugzilla.xamarin.com. Reason: Could not find SDK directory 'C:\Users\Adrián\AppData\Local\Android\android-sdk'.  Is --sdk-dir set appropriately? (XA0000) (StepCounter.Android)

The url inside is ok. its some kind of utf vs iso8859-1(5) translation inside xamarin?

This Xamarin locale issue is going to hit 1 year old, and no fix yet.

Thank you.
Comment 27 Brendan Zagaeski (Xamarin Support) 2014-08-14 01:32:16 UTC
Created attachment 7688 [details]
Test Case

A little update here to pinpoint the problem more precisely:

What's happening is that "Xamarin.Android.Build.Tasks" is writing to the standard input of `mandroid.exe` using UTF-8 encoding _without_ a unicode byte order mark, and then `mandroid.exe` is mistakenly reading those strings using iso-8859-1/latin1 encoding.

Changing "Xamarin.Android.Build.Tasks" so that it adds a byte order mark might be an acceptable solution.



## Steps to reproduce

1. Download the attached test case, and unzip it on Windows.


2. Attempt to launch the "AndroidAppé" project. The build fails before creating an `.apk`.


3. Update the `androidSdkPath` in the "App1" project to match the actual path of the Android SDK on your computer.


4. Build and run the "App1" project.



## Result

The "App1" project successfully creates the "AndroidAppé" APK file:
AndroidAppé\obj\Debug\android\bin\AndroidAppé.AndroidAppé.apk
Comment 28 Brendan Zagaeski (Xamarin Support) 2014-08-28 12:33:22 UTC
*** Bug 13475 has been marked as a duplicate of this bug. ***
Comment 29 Brendan Zagaeski (Xamarin Support) 2014-08-28 12:33:52 UTC
*** Bug 19323 has been marked as a duplicate of this bug. ***
Comment 30 dean.ellis 2014-09-15 06:06:28 UTC
*** Bug 22946 has been marked as a duplicate of this bug. ***
Comment 31 JEMB 2014-09-16 13:52:30 UTC
Hi guys,

This is a possible solution. 

1) I have moved the xamarin project directory to (for example) a directory without special characters: C:\Xamarin Studio.
2) Also I have moved the contents of Android folder (in appData of user directory) to same directory C:\Xamarin Studio\Android (the two folders android-sdk and ndk)
3) In Xamarin Studio go to Tools -> Options -> Projects -> SDK Locations for set the new directories in Android SDK and Android NDK

I hope can run in your projects...

Regards
Jesús
Comment 32 Brendan Zagaeski (Xamarin Support) 2014-11-12 22:15:35 UTC
One extra note just to be a bit redundant and state the obvious: any changes to this behavior will need to be carefully QA'd.

For example, bug 24479 should probably also be addressed during the same release cycle.

Some subset of build, activation, and license validation scenarios involving non-ASCII characters in file names, project names, project directory names, solution names, solution directory names, user names, computer names, and hard drive volume names might also need to be tested.
Comment 33 Brendan Zagaeski (Xamarin Support) 2014-12-01 20:02:50 UTC
Another closely related bug: bug 18897.
Comment 34 Jonathan Pryor 2015-02-27 13:19:04 UTC
This is, hilariously, Bug #23771 [0].

As part of its operation, Xamarin.Android.Build.Tasks.dll launches `mandroid.exe`, redirecting standard input, standard output, and standard error, sending commands and receiving output.

To support non-ASCII paths, Xamarin.Android.Build.Tasks.dll and mandroid.exe explicitly override the encoding of the standard input stream to use UTF-8 (because using Encoding.Default would be insane).

Furthermore, mandroid.exe bundles the mono runtime.

The problem is that the mono bundled with mandroid.exe has buggy UTF-8 decoding. Consequently, the Unicode-using path that Xamarin.Android.Build.Tasks.dll sends to mandroid.exe isn't decoded properly, resulting in Directory.Exists() stating that the SDK directory doesn't exist.

[0]: Modified test case from Bug #23771:

  using System;
  using System.Text;

  class App {
    public static void Main ()
    {
      var input   = "\u00e9";
      var encoded = Encoding.UTF8.GetBytes(input);
      var decoder = Encoding.UTF8.GetDecoder();
      var chars   = new char [1];
      var result  = new StringBuilder();
      var bytes   = new byte [1];
      foreach (var b in encoded) {
        bytes [0] = b;
        int bytesUsed, charsUsed;
        bool completed;
        decoder.Convert (bytes, 0, bytes.Length, chars, 0, chars.Length, false,
          out bytesUsed, out charsUsed, out completed);
        result.Append (chars, 0, charsUsed);
        Console.WriteLine (
            "# decoder.Convert: bytesUsed={0}; charsUsed={1}; " + 
            "completed={2}, result='{3}'",
            bytesUsed, charsUsed, completed, result);
      }
      var res = result.ToString ();
      Console.WriteLine (" input: '{0}' Length={1}", input, input.Length);
      Console.WriteLine ("result: '{0}' Length={1}", res, res.Length);
    }
  }

Execution output:

# decoder.Convert: bytesUsed=1; charsUsed=1; completed=True, result='�'
# decoder.Convert: bytesUsed=3; charsUsed=1; completed=False, result='��'
 input: 'é' Length=1
result: '��' Length=2
Comment 35 Brendan Zagaeski (Xamarin Support) 2015-03-06 14:15:51 UTC
One more closely related bug: bug 27735.

I'm adding that link just in case it'll be helpful to have all of the related bugs linked here for testing and verification once the fixed build from comment 34 is ready.
Comment 36 Jonathan Pryor 2015-03-09 11:40:27 UTC
*** Bug 27749 has been marked as a duplicate of this bug. ***
Comment 37 Jonathan Pryor 2015-04-10 17:40:29 UTC
*** Bug 28833 has been marked as a duplicate of this bug. ***
Comment 38 Jonathan Pryor 2015-07-09 12:09:17 UTC
*** Bug 27074 has been marked as a duplicate of this bug. ***
Comment 39 Jonathan Pryor 2015-07-20 12:12:19 UTC
*** Bug 27076 has been marked as a duplicate of this bug. ***
Comment 40 Jonathan Pryor 2015-10-06 17:34:24 UTC
*** Bug 34599 has been marked as a duplicate of this bug. ***
Comment 41 Jonathan Pryor 2015-10-16 11:09:14 UTC
*** Bug 34772 has been marked as a duplicate of this bug. ***
Comment 42 Brendan Zagaeski (Xamarin Support) 2015-11-23 14:47:52 UTC
*** Bug 35756 has been marked as a duplicate of this bug. ***
Comment 43 Brendan Zagaeski (Xamarin Support) 2016-02-10 23:54:00 UTC
*** Bug 38017 has been marked as a duplicate of this bug. ***
Comment 44 Tarcisio 2016-03-30 19:21:37 UTC
I 'm having the same problem , because of my username , several calls and plays do not find the directory
Comment 45 Jonathan Pryor 2016-06-01 14:12:07 UTC
*** Bug 34341 has been marked as a duplicate of this bug. ***
Comment 46 Jonathan Pryor 2016-06-27 14:27:22 UTC
This should be fixed in Cycle 8 (Xamarin.Android 6.2) because the mandroid daemon no longer exists. (And there was much rejoicing.)
Comment 47 Abhishek 2016-06-30 18:34:53 UTC
I have checked this issue and able to reproduce this issue with Xamarin.VisualStudio_3.9.344_e23ab728783843bbf61a5bf3e422fd4e6287ac58. I have created an Android Application and apply the code given in the comment 34.

Screencast: http://www.screencast.com/t/yNhkOlVV0YD
Application Output: https://gist.github.com/Abhishekk360/530e52fe01ab2a32971c302f0961a07e

To verify this issue I have checked this issue with latest master Build:
Xamarin.VisualStudio_99.0.0.2779_bea5c66e568c4938b7b9a767fabc13a6ee22995a. Now this issue is working fine. Here is the screencast for the same:
http://www.screencast.com/t/3NPKmMqENb5

Application Output: https://gist.github.com/Abhishekk360/2685043d14bcdce2e5791a95a099edb9

Thanks!
Comment 48 Jonathan Pryor 2016-06-30 18:52:13 UTC
@Abhishek: I believe that you've misconstrued the issue.

Comment #34 was a *cause*, buried within the mandroid daemon. Whether or not Comment #34 is "fixed" or not isn't relevant to testing and verifying the bug itself.

The bug is simple: Create a new project into a directory that contains non-ASCII characters, then build and deploy it. Thus:

That's it.

The app itself doesn't need to have non-ASCII characters (that might be separate bug), but the *directory it's in* needs to have non-ASCII characters.

For example, in Comment #0 the Android SDK was installed into C:\Users\André, and the created project was also within C:\Users\André. Attempting to build in such an environment resulted in XA0000 errors.
Comment 49 Abhishek 2016-07-01 11:41:38 UTC
Thanks @ Jonathan,

I have checked this issue and able to reproduce this issue with Xamarin.VisualStudio_3.9.344_e23ab728783843bbf61a5bf3e422fd4e6287ac58. As per comment 48, I am getting build error. Here is the screencast for the same: http://www.screencast.com/t/upKlX28JoVo

Build Output: https://gist.github.com/Abhishekk360/9c0f74240943f502b90d69df0737b9b1

To verify this issue I have checked this issue with latest master Build:
Xamarin.VisualStudio_99.0.0.2779_bea5c66e568c4938b7b9a767fabc13a6ee22995a. Now this issue is working fine. Here is the screencast for the same: http://www.screencast.com/t/lO5Uf2ZY

Build Output: https://gist.github.com/Abhishekk360/763041e5c3ce6c462bb0190263b464c3
Application Output: https://gist.github.com/Abhishekk360/39ecf7db48c2f114dbbc70397304e599
Screencast: http://www.screencast.com/t/lO5Uf2ZY

Hence closing this issue.

Thanks!
Comment 50 Naqeeb 2016-07-04 13:46:48 UTC
As per comment 49 closing this issue.
Comment 51 Adam Hartley [MSFT] 2017-06-28 12:02:12 UTC
*** Bug 37339 has been marked as a duplicate of this bug. ***