Bug 56489 - AOT compiler does not warn about wrong C#/IL code
Summary: AOT compiler does not warn about wrong C#/IL code
Alias: None
Product: iOS
Classification: Xamarin
Component: Mono runtime / AOT compiler (show other bugs)
Version: XI 10.8 (d15-1)
Hardware: All Mac OS
: --- enhancement
Target Milestone: Future Cycle (TBD)
Assignee: Rolf Bjarne Kvinge [MSFT]
Depends on: 59634
  Show dependency tree
Reported: 2017-05-16 11:04 UTC by Slava Osipov
Modified: 2017-09-21 11:19 UTC (History)
5 users (show)

See Also:
Is this bug a regression?: ---
Last known good build:

Sample project (202.75 KB, application/x-zip-compressed)
2017-05-16 11:04 UTC, Slava Osipov
crash report (32.73 KB, text/plain)
2017-05-17 11:01 UTC, Slava Osipov
Build log (186.48 KB, application/x-zip-compressed)
2017-05-17 11:16 UTC, Slava Osipov
Second sample (183.87 KB, application/x-zip-compressed)
2017-05-18 15:08 UTC, Slava Osipov
Sample2 build log (184.77 KB, application/x-zip-compressed)
2017-05-19 11:21 UTC, Slava Osipov
build and debug logs (187.36 KB, application/x-zip-compressed)
2017-05-19 15:16 UTC, Slava Osipov
Mac build (803.66 KB, application/x-zip-compressed)
2017-05-22 10:29 UTC, Slava Osipov
no device-specific (24.96 KB, text/plain)
2017-05-22 11:07 UTC, Slava Osipov

Description Slava Osipov 2017-05-16 11:04:53 UTC
Created attachment 22193 [details]
Sample project

I faced unexpected behavior of AOT compiler when resulting generated code crashes my application on iOS.
1) Use attached sample project solution and open it in Visual Studio
2) Connect real iOS device to build Mac machine
3) Start debugging of CrossTest.iOS project in Debug/iPhone configuration on connected iOS device
4) Wait for NullReferenceException.

---My environment details:
Visual Studio 2015 Enterprise Update 3
  - Xamarin 4.4

iOS device: iPad 2.0 iOS 9.3.5
   - Model MC981B/A

Build Machine: Mac Mini (Late 2012) 
- OS X El Captain (10.11.6)
- Xamarin Studio 6.3 (build 863)
   - Runtime:
     - Mono 4.8.0 (mono-4.8.0-brach/9d74414)(64 bit)
     - GTK+ 2.24.23
   - Xamarin.iOS
     - Version
     - Hash a04678c2
     - Branch d15-1
- Xcode 8.2.1 (11766.1) Build 8C1002

---Crash details:
 	0xFFFFFFFFFFFFFFFF in DocumentFormat.OpenXml.Packaging.OpenSettings..ctor	C#
>	0x2C in CrossTest.MyClass.Connect at ..\CrossTest\CrossTest\CrossTest\MyClass.cs:28,13	C#
 	0xF in CrossTest.iOS.ViewController.ViewDidLoad at ..\CrossTest\CrossTest\CrossTest.iOS\ViewController.cs:19,13	C#
 	0xFFFFFFFFFFFFFFFF in UIKit.UIApplication.UIApplicationMain	C#
 	0xB in UIKit.UIApplication.Main at /Users/builder/data/lanes/4466/a04678c2/source/xamarin-macios/src/UIKit/UIApplication.cs:79,4	C#
 	0x3B in UIKit.UIApplication.Main at /Users/builder/data/lanes/4466/a04678c2/source/xamarin-macios/src/UIKit/UIApplication.cs:63,4	C#
 	0x8 in CrossTest.iOS.Application.Main at ..\CrossTest\CrossTest\CrossTest.iOS\Main.cs:17,4	C#

NRE occured in constructor of OpenSettings class ( https://github.com/OfficeDev/Open-XML-SDK/blob/V2.6/DocumentFormat.OpenXml/src/Framework/OpenXmlPackage.cs#L2255 )

--- Actual Results:
iOS project compiled and run with crash

--- Expected Results:
iOS project not compiled with meaningful error message(s)

Additional Info: 
1) I have not faced this type of crash in iOS Simulator.
2) In my real project NRE occured in different places within Open-XML-SDK assembly.
3) It is possible that different iOS device will cause NRE in different place, I can't test.

--- Notes:
1) I know that Open-Xml-SDK referenced by my project is not designed to be used on iOS
2) It is possible that Open-XML-SDK does not meet AOT limitations
3) My best guess - mtouch generates wrong native code/structures
Comment 1 Vincent Dondain [MSFT] 2017-05-16 17:47:50 UTC
I could not reproduce the NullReferenceException with the following environment: https://gist.github.com/VincentDondain/91b1d7565cb379f5dbc96aa1996da459

This is basically the stable versions of our products as of today.

I tried on an iPhone 7 device running iOS 10.3 (I do not have older devices with me to try out).

Now as for your expected results:

"iOS project not compiled with meaningful error message(s)"

This is a runtime error not a build error, there's no way for us to anticipate that (:

@slava, could you please try the stable versions of our products and provide us a full verbose build log?

On Visual Studio for Windows set the build verbosity to diagnostic: Tools > Options > Projects and Solutions > Build and Run.

It would also greatly help to get a symbolicated and complete crash log.

Note: as Sebastien pointed out to me AOT limitations generally throw ExecutionEngineException, not NRE.
Comment 2 Zoltan Varga 2017-05-16 17:50:34 UTC
Also, could you try disabling the linker in the project settings, and see whenever that fixes the problem ?
Comment 3 Slava Osipov 2017-05-17 10:21:33 UTC
hi Zoltan,
I'm not sure what did you mean by "disabling the linker". Assigning 'Linker Behavior' to 'Don't Link' just causes expected compilation error:
1>C:\Program Files (x86)\MSBuild\Xamarin\iOS\Xamarin.iOS.Common.targets(747,3): error : This version of Xamarin.iOS requires the iOS 10.3 SDK (shipped with Xcode 8.3). Either upgrade Xcode to get the required header files or set the managed linker behaviour to Link Framework SDKs Only (to try to avoid the new APIs).
Comment 4 Slava Osipov 2017-05-17 10:32:57 UTC
hi Vincent,
I've updated Xamarin to 4.5 in my Visual Studio on Windows PC and updated Xamarin Studio on my Mac-mini to the latest stable version too.
Issue still reproduced.

> This is a runtime error not a build error
I can not agree. If you will check source code (I've provided link) you will see that OpenSettings class hasn't explicitly declared constructor, so empty constructor injected implicitly. I have no idea how it can be a runtime problem if execution of "empty" constructor causes NRE.
Comment 5 Slava Osipov 2017-05-17 11:01:44 UTC
Created attachment 22225 [details]
crash report

Crash report generated using latest stable Xamarin software.
Comment 6 Slava Osipov 2017-05-17 11:16:00 UTC
Created attachment 22226 [details]
Build log

Build log with removed sensitive data.
Comment 7 Vincent Dondain [MSFT] 2017-05-18 12:02:09 UTC
Yes what Zoltan is recommending is to set the linker to "Don't link" (:

Because you're using a Xamarin.iOS version that requires the iOS 10.3 SDK you can either upgrade your Xcode or downgrade your Xamarin.iOS version to be able to set the linker to "Don't link" (otherwise using the linker is a temporary way to strip the new APIs).

In this case @slava please upgrade Xcode to 8.3, set the linker to "Don't link" and let us know if that fixes the issue?
Comment 8 Slava Osipov 2017-05-18 13:30:09 UTC
I'm updating my Mac software. I will provide results soon.
Comment 9 Slava Osipov 2017-05-18 14:04:45 UTC
I can confirm that crash is no longer reproduced with latest Mac&Xamarin software on this sample project in my environment. But in my main project issue still present. I will prepare new sample project.
Comment 10 Slava Osipov 2017-05-18 15:08:15 UTC
Created attachment 22271 [details]
Second sample
Comment 11 Slava Osipov 2017-05-18 15:12:25 UTC
Using second sample ussue probably will not be reproduced on first run.
1) Run app from Visual Studio under debug
2) Tap button 'Test' in app
3) NRE may occur in PresentationDocument.Open method
4) Stop debug
5) Repeat step 1 - actual NRE should occur somewhere in XF
Comment 12 Slava Osipov 2017-05-18 15:14:39 UTC
Making changes in PCL project (just to be sure that it will be rebuilt) we may face actual NRE on first start.
Comment 13 Vincent Dondain [MSFT] 2017-05-19 10:47:51 UTC
I still can't reproduce this with an all stable (Mac) environment: https://gist.github.com/VincentDondain/bdc79e1cfdb63952a832d5ed59a7736b

I followed carefully the steps in comment 11, ran multiple times, nothing.

You said that "that crash is no longer reproduced with latest Mac&Xamarin software on [the original] sample project", what exactly did you change?

Can you give us your full version information as well as full build log for the issue with the new sample?
Comment 14 Slava Osipov 2017-05-19 11:05:56 UTC
hi Vincent
> what exactly did you change?
I've updated:
1) OS from Mac OS X El Capitan to macOS Sierra
2) Xcode from 8.2.1 to 8.3.2
3) Changed 'Linker Behavior' to 'Don't link' in project settings and written new code for Sample2 project.
4) Additionally tested on iPad 4 (Model MD514B/A)

Environment information - https://gist.github.com/slava-osipov-qs/59f1269016dcff086bf6fef3a222803d

iOS devices: 
iPad 2 (MC981B/A) iOS 9.3.5
iPad 4 (MD514B/A) iOS 10.3.2
Comment 15 Vincent Dondain [MSFT] 2017-05-19 11:20:36 UTC
Ok so sample 1 now works but now sample 2 is broken.

Did you try to disable the linker when running sample 2 on device?

For the Debug | iPhone configuration. The sample shows it's not.
Comment 16 Slava Osipov 2017-05-19 11:21:45 UTC
Created attachment 22308 [details]
Sample2 build log
Comment 17 Slava Osipov 2017-05-19 11:24:39 UTC
>Did you try to disable the linker when running sample 2 on device?
>For the Debug | iPhone configuration. The sample shows it's not.
Probably it is the UI problem. It shows me 'Don't link'
I will retest again
Comment 18 Vincent Dondain [MSFT] 2017-05-19 11:32:11 UTC
This build log shows you're using the linker in SdkOnly mode.
Comment 19 Slava Osipov 2017-05-19 11:39:08 UTC
I've ensured that Debug/iPhone configuration has '<MtouchLink>None</MtouchLink>' section. App generates NRE in the constructor of OpenSettings class now - https://ibb.co/jYy9kv and debug log is - https://gist.github.com/slava-osipov-qs/4b2628cb481a4d4927b2f98a04b72147
Comment 20 Vincent Dondain [MSFT] 2017-05-19 14:39:18 UTC
@slava can you please also include your build log for your last attempt when the linker is disabled?

Must must have some different settings that prevents me from reproducing and isolating the issue.
Comment 21 Slava Osipov 2017-05-19 15:16:19 UTC
Created attachment 22312 [details]
build and debug logs
Comment 22 Slava Osipov 2017-05-19 15:19:05 UTC
I've run app on iPad 2 (MC981B/A)
It has wrong timezone settings, so build environment and iPad has 1 hour difference.
Comment 23 Vincent Dondain [MSFT] 2017-05-19 15:22:54 UTC
Thanks for the logs. Any chance you can try this sample on Visual Studio for Mac (instead of Windows).

I wonder if it's not a windows specific issue.
Comment 24 Slava Osipov 2017-05-22 10:29:37 UTC
Created attachment 22344 [details]
Mac build

Bundle name was changed, I guess it should not be a problem.
Comment 25 Slava Osipov 2017-05-22 11:07:21 UTC
Created attachment 22346 [details]
no device-specific

I've tried to set/unset 'Enable device-specific builds' and can see different crashes when I built app on Mac. Debug log when this option is switched off.
Comment 26 Slava Osipov 2017-05-22 14:04:02 UTC
hi Vincent.
I can see interesting in stack trace. See attach in comment #21
In debug.txt we can see two same lines in user code:
>2017-05-19 21:10:45.320 Sample2iOS[1092:2939994] critical: 	8   Sample2iOS                          0x0407d247 handle_signal_exception + 42
>2017-05-19 21:10:45.321 Sample2iOS[1092:2939994] critical: 	9   Sample2iOS                          0x02776eac Sample2_Main_Button_OnClicked_object_System_EventArgs + 1000
>2017-05-19 21:10:45.321 Sample2iOS[1092:2939994] critical: 	10  Sample2iOS                          0x02776eac Sample2_Main_Button_OnClicked_object_System_EventArgs + 1000
I'm not sure what does it mean exactly. Probably it is the effect of stack damaging. Or it may mean inlining of code for OpenSettings constructor. 
I've seen same duplication when built app in Mac environment, just with different offset.
Comment 27 Slava Osipov 2017-05-24 06:16:02 UTC
hi Vincent.
It seems that my build environment is different to your. I've tried to build my main project on other Mac and issue was not reproduced, like in your environment. Also I've removed OpenXML-SDK dependency from my main project and issue was not reproduced after building in my build environment. So I guess combination of build environment and OpenXML-SDK dependency causes this issue.
It is possible that mtouch depends on some environment variables or something else, but I don't know how it depends. If you can request from me information about my build environment, please request, I will be happy to collect this information for you. Probably this will help to find a root of problem.
Comment 28 Rolf Bjarne Kvinge [MSFT] 2017-05-24 15:26:04 UTC
I can reproduce in 32-bit.

Crash report: https://gist.github.com/rolfbjarne/f078f06585c1a03ac485b0f751513e91

Attaching lldb shows this backtrace:

(lldb) bt
* thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xe59fc000)
  * frame #0: 0x0246d560 Sample2iOS`System_Collections_Generic_List_1_T_INST_Add_T_INST(this=0x05f206a0, item=158671488) at list.cs:228
    frame #1: 0x00e26764 Sample2iOS`Xamarin_Forms_Xaml_XamlParser_ParseXamlAttributes_System_Xml_XmlReader_System_Collections_Generic_IList_1_System_Collections_Generic_KeyValuePair_2_string_string_(reader=0x05f185d8, xmlns=94572956) at <unknown>:1
    frame #2: 0x00e23bac Sample2iOS`Xamarin_Forms_Xaml_XamlParser_ParseXaml_Xamarin_Forms_Xaml_RootNode_System_Xml_XmlReader(rootNode=0x05f20278, reader=0x05f185d8) at <unknown>:1
    frame #3: 0x00df9ab4 Sample2iOS`Xamarin_Forms_Xaml_XamlLoader_Load_object_string(view=0x05f4b848, xaml=0x05f10918) at <unknown>:1
    frame #4: 0x00df972c Sample2iOS`Xamarin_Forms_Xaml_XamlLoader_Load_object_System_Type(view=0x05f4b848, callingType=0x09775d70) at <unknown>:1
    frame #5: 0x00df93ac Sample2iOS`Xamarin_Forms_Xaml_Extensions_LoadFromXaml_TXaml_REF_TXaml_REF_System_Type(view=99924040, callingType=0x09775d70) at <unknown>:1
    frame #6: 0x00def638 Sample2iOS`Sample2_App_InitializeComponent(this=0x05f4b848) at Sample2.App.xaml.g.cs:19
    frame #7: 0x00def128 Sample2iOS`Sample2_App__ctor(this=0x05f4b848) at App.xaml.cs:14
    frame #8: 0x0001f7f0 Sample2iOS`Sample2_iOS_AppDelegate_FinishedLaunching_UIKit_UIApplication_Foundation_NSDictionary(this=0x05f26958, app=0x05f31260, options=0x00000000) at AppDelegate.cs:26
    frame #9: 0x0029d8b8 Sample2iOS`wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr + 256
    frame #10: 0x025efda8 Sample2iOS`mono_jit_runtime_invoke(method=0x0683abd8, obj=<unavailable>, params=0x00000000, exc=<unavailable>, error=<unavailable>) at mini-runtime.c:2509 [opt]
    frame #11: 0x0263be2a Sample2iOS`do_runtime_invoke(method=0x0683abc0, obj=0x05f26958, params=0x05a31824, exc=0x00000000, error=<unavailable>) at object.c:2860 [opt]
    frame #12: 0x0263bdc0 Sample2iOS`mono_runtime_invoke [inlined] mono_runtime_invoke_checked(method=<unavailable>, obj=0x05f26958, params=0x05a31824, error=0x05a31734) at object.c:3018 [opt]
    frame #13: 0x0263bd94 Sample2iOS`mono_runtime_invoke(method=0x0683abc0, obj=0x05f26958, params=0x05a31824, exc=<unavailable>) at object.c:2917 [opt]
    frame #14: 0x00009052 Sample2iOS`native_to_managed_trampoline_7(self=0x05b57190, _cmd="application:didFinishLaunchingWithOptions:", managed_method_ptr=0x028416cc, p0=0x05b55f90, p1=0x00000000, token_ref=768) at registrar.m:320
    frame #15: 0x0000963a Sample2iOS`::-[AppDelegate application:didFinishLaunchingWithOptions:](self=0x05b57190, _cmd="application:didFinishLaunchingWithOptions:", p0=0x05b55f90, p1=0x00000000) at registrar.m:6033
Comment 29 Vincent Dondain [MSFT] 2017-05-24 15:39:04 UTC
I could confirm this too in 32 bit mode with the following environment: https://gist.github.com/VincentDondain/250ebc14c90b77e599b86f18960ede95

Marking the bug as confirmed as all information was provided (:

Comment 30 Slava Osipov 2017-05-26 10:24:47 UTC
hi Vincent.
Do you have any ideas about ETA for this issue fix?
Comment 31 Vincent Dondain [MSFT] 2017-05-26 13:42:47 UTC
Hi Slava,

We now have all the information to reproduce this bug, it's assigned to the right component as well as the right person (Zoltan).

The target milestone is currently marked as 15.3 which is our next major release (currently in preview on our alpha channel).

You can expect to be notified of the advancement here and, if we have a fix, we'll let you know how to get it ASAP.
Comment 34 Zoltan Varga 2017-05-30 19:16:10 UTC
Seems like a dup of:

i.e. the app is too large, the executable in debug mode is 80mb. Does this also happen in release mode ?
Comment 35 Rolf Bjarne Kvinge [MSFT] 2017-06-01 15:50:17 UTC
The "Second sample" project works fine in release mode.

For some reason the debug build is 87MB, while the release build is 24MB (linker settings are identical).
Comment 36 Rolf Bjarne Kvinge [MSFT] 2017-06-01 18:11:13 UTC
I got the debug build down to 41mb by stripping the executable [1], and the problem still occurs.

The runtime behaviour may differ even without rebuilding, just tapping on the app I see crashes in different places (sometimes at startup, sometimes I have to tap "Test" to make it crash).

[1] This is required to strip debug builds: https://github.com/xamarin/xamarin-macios/pull/2160
Comment 37 Rolf Bjarne Kvinge [MSFT] 2017-06-01 18:32:34 UTC
If I enable the linker for all assemblies, the executable size drops to 34MB and the crash goes away.

This may very well be a problem with DocumentFormat.OpenXml.dll, it's an impressive assembly with over 64000 methods (~41000 after linking all assemblies).

This all indicates that it's a variation of bug #1102.

It might be possible to work around the problem by disabling debug info for this particular assembly (we don't currently support disabling debug info for specific assemblies, so I'd have to implement it).

However this is not something we'll be able to implement for 15.3, so I'm bumping the milestone.
Comment 38 Slava Osipov 2017-06-02 11:54:40 UTC
hi Rolf
I've used changes mentioned in #36 and this unblocked me. Thanks a lot.

But this issue still not solved. I expect meaningful error message at build time, if it is not possible to build correct binaries due to some limitations of a tool in tools chain.
Comment 39 Rolf Bjarne Kvinge [MSFT] 2017-09-21 08:45:48 UTC
(In reply to Rolf Bjarne Kvinge [MSFT] from comment #37)
> If I enable the linker for all assemblies, the executable size drops to 34MB
> and the crash goes away.
> This may very well be a problem with DocumentFormat.OpenXml.dll, it's an
> impressive assembly with over 64000 methods (~41000 after linking all
> assemblies).
> This all indicates that it's a variation of bug #1102.
> It might be possible to work around the problem by disabling debug info for
> this particular assembly (we don't currently support disabling debug info
> for specific assemblies, so I'd have to implement it).

To make things less confusing here, I've filed bug #59634 to track this.
Comment 40 Slava Osipov 2017-09-21 10:55:18 UTC
hi Rolf,
That is great that debug info injection will be optional per each assembly. So  problem mentioned in #1102 could be solved.
But for this ticket I would prefer to have meaningful error message that indicates inability to build proper binaries due to 'Section too large' limitation. I.e. I would prefer to stop build process with error message instead of reporting about successful build and further crash at run-time.
Comment 41 Rolf Bjarne Kvinge [MSFT] 2017-09-21 11:19:25 UTC
Unfortunately detecting this scenario is quite complicated. The proper place to detect this would be in the native linker (ld), which Apple ships. This would involve tracking down the problem inside ld in the first place, come up with a patch, tell Apple about the patch, and hopefully they'd accept it and release it (and it would probably not be released until a year later, with the next major Xcode release). And since it only happens with 32-bit code, and Apple is dropping 32-bit quickly, I'm not even sure they'd care enough to get the patch in.

Note You need to log in before you can comment on or make changes to this bug.