Created attachment 22193 [details]
I faced unexpected behavior of AOT compiler when resulting generated code crashes my application on iOS.
1) Use attached sample project solution and open it in Visual Studio
2) Connect real iOS device to build Mac machine
3) Start debugging of CrossTest.iOS project in Debug/iPhone configuration on connected iOS device
4) Wait for NullReferenceException.
---My environment details:
Visual Studio 2015 Enterprise Update 3
- Xamarin 4.4
iOS device: iPad 2.0 iOS 9.3.5
- Model MC981B/A
Build Machine: Mac Mini (Late 2012)
- OS X El Captain (10.11.6)
- Xamarin Studio 6.3 (build 863)
- Mono 4.8.0 (mono-4.8.0-brach/9d74414)(64 bit)
- GTK+ 2.24.23
- Version 10.8.0.175
- Hash a04678c2
- Branch d15-1
- Xcode 8.2.1 (11766.1) Build 8C1002
0xFFFFFFFFFFFFFFFF in DocumentFormat.OpenXml.Packaging.OpenSettings..ctor C#
> 0x2C in CrossTest.MyClass.Connect at ..\CrossTest\CrossTest\CrossTest\MyClass.cs:28,13 C#
0xF in CrossTest.iOS.ViewController.ViewDidLoad at ..\CrossTest\CrossTest\CrossTest.iOS\ViewController.cs:19,13 C#
0xFFFFFFFFFFFFFFFF in UIKit.UIApplication.UIApplicationMain C#
0xB in UIKit.UIApplication.Main at /Users/builder/data/lanes/4466/a04678c2/source/xamarin-macios/src/UIKit/UIApplication.cs:79,4 C#
0x3B in UIKit.UIApplication.Main at /Users/builder/data/lanes/4466/a04678c2/source/xamarin-macios/src/UIKit/UIApplication.cs:63,4 C#
0x8 in CrossTest.iOS.Application.Main at ..\CrossTest\CrossTest\CrossTest.iOS\Main.cs:17,4 C#
NRE occured in constructor of OpenSettings class ( https://github.com/OfficeDev/Open-XML-SDK/blob/V2.6/DocumentFormat.OpenXml/src/Framework/OpenXmlPackage.cs#L2255 )
--- Actual Results:
iOS project compiled and run with crash
--- Expected Results:
iOS project not compiled with meaningful error message(s)
1) I have not faced this type of crash in iOS Simulator.
2) In my real project NRE occured in different places within Open-XML-SDK assembly.
3) It is possible that different iOS device will cause NRE in different place, I can't test.
1) I know that Open-Xml-SDK referenced by my project is not designed to be used on iOS
2) It is possible that Open-XML-SDK does not meet AOT limitations
3) My best guess - mtouch generates wrong native code/structures
I could not reproduce the NullReferenceException with the following environment: https://gist.github.com/VincentDondain/91b1d7565cb379f5dbc96aa1996da459
This is basically the stable versions of our products as of today.
I tried on an iPhone 7 device running iOS 10.3 (I do not have older devices with me to try out).
Now as for your expected results:
"iOS project not compiled with meaningful error message(s)"
This is a runtime error not a build error, there's no way for us to anticipate that (:
@slava, could you please try the stable versions of our products and provide us a full verbose build log?
On Visual Studio for Windows set the build verbosity to diagnostic: Tools > Options > Projects and Solutions > Build and Run.
It would also greatly help to get a symbolicated and complete crash log.
Note: as Sebastien pointed out to me AOT limitations generally throw ExecutionEngineException, not NRE.
Also, could you try disabling the linker in the project settings, and see whenever that fixes the problem ?
I'm not sure what did you mean by "disabling the linker". Assigning 'Linker Behavior' to 'Don't Link' just causes expected compilation error:
1>C:\Program Files (x86)\MSBuild\Xamarin\iOS\Xamarin.iOS.Common.targets(747,3): error : This version of Xamarin.iOS requires the iOS 10.3 SDK (shipped with Xcode 8.3). Either upgrade Xcode to get the required header files or set the managed linker behaviour to Link Framework SDKs Only (to try to avoid the new APIs).
I've updated Xamarin to 4.5 in my Visual Studio on Windows PC and updated Xamarin Studio on my Mac-mini to the latest stable version too.
Issue still reproduced.
> This is a runtime error not a build error
I can not agree. If you will check source code (I've provided link) you will see that OpenSettings class hasn't explicitly declared constructor, so empty constructor injected implicitly. I have no idea how it can be a runtime problem if execution of "empty" constructor causes NRE.
Created attachment 22225 [details]
Crash report generated using latest stable Xamarin software.
Created attachment 22226 [details]
Build log with removed sensitive data.
Yes what Zoltan is recommending is to set the linker to "Don't link" (:
Because you're using a Xamarin.iOS version that requires the iOS 10.3 SDK you can either upgrade your Xcode or downgrade your Xamarin.iOS version to be able to set the linker to "Don't link" (otherwise using the linker is a temporary way to strip the new APIs).
In this case @slava please upgrade Xcode to 8.3, set the linker to "Don't link" and let us know if that fixes the issue?
I'm updating my Mac software. I will provide results soon.
I can confirm that crash is no longer reproduced with latest Mac&Xamarin software on this sample project in my environment. But in my main project issue still present. I will prepare new sample project.
Created attachment 22271 [details]
Using second sample ussue probably will not be reproduced on first run.
1) Run app from Visual Studio under debug
2) Tap button 'Test' in app
3) NRE may occur in PresentationDocument.Open method
4) Stop debug
5) Repeat step 1 - actual NRE should occur somewhere in XF
Making changes in PCL project (just to be sure that it will be rebuilt) we may face actual NRE on first start.
I still can't reproduce this with an all stable (Mac) environment: https://gist.github.com/VincentDondain/bdc79e1cfdb63952a832d5ed59a7736b
I followed carefully the steps in comment 11, ran multiple times, nothing.
You said that "that crash is no longer reproduced with latest Mac&Xamarin software on [the original] sample project", what exactly did you change?
Can you give us your full version information as well as full build log for the issue with the new sample?
> what exactly did you change?
1) OS from Mac OS X El Capitan to macOS Sierra
2) Xcode from 8.2.1 to 8.3.2
3) Changed 'Linker Behavior' to 'Don't link' in project settings and written new code for Sample2 project.
4) Additionally tested on iPad 4 (Model MD514B/A)
Environment information - https://gist.github.com/slava-osipov-qs/59f1269016dcff086bf6fef3a222803d
iPad 2 (MC981B/A) iOS 9.3.5
iPad 4 (MD514B/A) iOS 10.3.2
Ok so sample 1 now works but now sample 2 is broken.
Did you try to disable the linker when running sample 2 on device?
For the Debug | iPhone configuration. The sample shows it's not.
Created attachment 22308 [details]
Sample2 build log
>Did you try to disable the linker when running sample 2 on device?
>For the Debug | iPhone configuration. The sample shows it's not.
Probably it is the UI problem. It shows me 'Don't link'
I will retest again
This build log shows you're using the linker in SdkOnly mode.
I've ensured that Debug/iPhone configuration has '<MtouchLink>None</MtouchLink>' section. App generates NRE in the constructor of OpenSettings class now - https://ibb.co/jYy9kv and debug log is - https://gist.github.com/slava-osipov-qs/4b2628cb481a4d4927b2f98a04b72147
@slava can you please also include your build log for your last attempt when the linker is disabled?
Must must have some different settings that prevents me from reproducing and isolating the issue.
Created attachment 22312 [details]
build and debug logs
I've run app on iPad 2 (MC981B/A)
It has wrong timezone settings, so build environment and iPad has 1 hour difference.
Thanks for the logs. Any chance you can try this sample on Visual Studio for Mac (instead of Windows).
I wonder if it's not a windows specific issue.
Created attachment 22344 [details]
Bundle name was changed, I guess it should not be a problem.
Created attachment 22346 [details]
I've tried to set/unset 'Enable device-specific builds' and can see different crashes when I built app on Mac. Debug log when this option is switched off.
I can see interesting in stack trace. See attach in comment #21
In debug.txt we can see two same lines in user code:
>2017-05-19 21:10:45.320 Sample2iOS[1092:2939994] critical: 8 Sample2iOS 0x0407d247 handle_signal_exception + 42
>2017-05-19 21:10:45.321 Sample2iOS[1092:2939994] critical: 9 Sample2iOS 0x02776eac Sample2_Main_Button_OnClicked_object_System_EventArgs + 1000
>2017-05-19 21:10:45.321 Sample2iOS[1092:2939994] critical: 10 Sample2iOS 0x02776eac Sample2_Main_Button_OnClicked_object_System_EventArgs + 1000
I'm not sure what does it mean exactly. Probably it is the effect of stack damaging. Or it may mean inlining of code for OpenSettings constructor.
I've seen same duplication when built app in Mac environment, just with different offset.
It seems that my build environment is different to your. I've tried to build my main project on other Mac and issue was not reproduced, like in your environment. Also I've removed OpenXML-SDK dependency from my main project and issue was not reproduced after building in my build environment. So I guess combination of build environment and OpenXML-SDK dependency causes this issue.
It is possible that mtouch depends on some environment variables or something else, but I don't know how it depends. If you can request from me information about my build environment, please request, I will be happy to collect this information for you. Probably this will help to find a root of problem.
I can reproduce in 32-bit.
Crash report: https://gist.github.com/rolfbjarne/f078f06585c1a03ac485b0f751513e91
Attaching lldb shows this backtrace:
* thread #1, name = 'tid_403', queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=1, address=0xe59fc000)
* frame #0: 0x0246d560 Sample2iOS`System_Collections_Generic_List_1_T_INST_Add_T_INST(this=0x05f206a0, item=158671488) at list.cs:228
frame #1: 0x00e26764 Sample2iOS`Xamarin_Forms_Xaml_XamlParser_ParseXamlAttributes_System_Xml_XmlReader_System_Collections_Generic_IList_1_System_Collections_Generic_KeyValuePair_2_string_string_(reader=0x05f185d8, xmlns=94572956) at <unknown>:1
frame #2: 0x00e23bac Sample2iOS`Xamarin_Forms_Xaml_XamlParser_ParseXaml_Xamarin_Forms_Xaml_RootNode_System_Xml_XmlReader(rootNode=0x05f20278, reader=0x05f185d8) at <unknown>:1
frame #3: 0x00df9ab4 Sample2iOS`Xamarin_Forms_Xaml_XamlLoader_Load_object_string(view=0x05f4b848, xaml=0x05f10918) at <unknown>:1
frame #4: 0x00df972c Sample2iOS`Xamarin_Forms_Xaml_XamlLoader_Load_object_System_Type(view=0x05f4b848, callingType=0x09775d70) at <unknown>:1
frame #5: 0x00df93ac Sample2iOS`Xamarin_Forms_Xaml_Extensions_LoadFromXaml_TXaml_REF_TXaml_REF_System_Type(view=99924040, callingType=0x09775d70) at <unknown>:1
frame #6: 0x00def638 Sample2iOS`Sample2_App_InitializeComponent(this=0x05f4b848) at Sample2.App.xaml.g.cs:19
frame #7: 0x00def128 Sample2iOS`Sample2_App__ctor(this=0x05f4b848) at App.xaml.cs:14
frame #8: 0x0001f7f0 Sample2iOS`Sample2_iOS_AppDelegate_FinishedLaunching_UIKit_UIApplication_Foundation_NSDictionary(this=0x05f26958, app=0x05f31260, options=0x00000000) at AppDelegate.cs:26
frame #9: 0x0029d8b8 Sample2iOS`wrapper_runtime_invoke_object_runtime_invoke_dynamic_intptr_intptr_intptr_intptr + 256
frame #10: 0x025efda8 Sample2iOS`mono_jit_runtime_invoke(method=0x0683abd8, obj=<unavailable>, params=0x00000000, exc=<unavailable>, error=<unavailable>) at mini-runtime.c:2509 [opt]
frame #11: 0x0263be2a Sample2iOS`do_runtime_invoke(method=0x0683abc0, obj=0x05f26958, params=0x05a31824, exc=0x00000000, error=<unavailable>) at object.c:2860 [opt]
frame #12: 0x0263bdc0 Sample2iOS`mono_runtime_invoke [inlined] mono_runtime_invoke_checked(method=<unavailable>, obj=0x05f26958, params=0x05a31824, error=0x05a31734) at object.c:3018 [opt]
frame #13: 0x0263bd94 Sample2iOS`mono_runtime_invoke(method=0x0683abc0, obj=0x05f26958, params=0x05a31824, exc=<unavailable>) at object.c:2917 [opt]
frame #14: 0x00009052 Sample2iOS`native_to_managed_trampoline_7(self=0x05b57190, _cmd="application:didFinishLaunchingWithOptions:", managed_method_ptr=0x028416cc, p0=0x05b55f90, p1=0x00000000, token_ref=768) at registrar.m:320
frame #15: 0x0000963a Sample2iOS`::-[AppDelegate application:didFinishLaunchingWithOptions:](self=0x05b57190, _cmd="application:didFinishLaunchingWithOptions:", p0=0x05b55f90, p1=0x00000000) at registrar.m:6033
I could confirm this too in 32 bit mode with the following environment: https://gist.github.com/VincentDondain/250ebc14c90b77e599b86f18960ede95
Marking the bug as confirmed as all information was provided (:
Do you have any ideas about ETA for this issue fix?
We now have all the information to reproduce this bug, it's assigned to the right component as well as the right person (Zoltan).
The target milestone is currently marked as 15.3 which is our next major release (currently in preview on our alpha channel).
You can expect to be notified of the advancement here and, if we have a fix, we'll let you know how to get it ASAP.
Seems like a dup of:
i.e. the app is too large, the executable in debug mode is 80mb. Does this also happen in release mode ?
The "Second sample" project works fine in release mode.
For some reason the debug build is 87MB, while the release build is 24MB (linker settings are identical).
I got the debug build down to 41mb by stripping the executable , and the problem still occurs.
The runtime behaviour may differ even without rebuilding, just tapping on the app I see crashes in different places (sometimes at startup, sometimes I have to tap "Test" to make it crash).
 This is required to strip debug builds: https://github.com/xamarin/xamarin-macios/pull/2160
If I enable the linker for all assemblies, the executable size drops to 34MB and the crash goes away.
This may very well be a problem with DocumentFormat.OpenXml.dll, it's an impressive assembly with over 64000 methods (~41000 after linking all assemblies).
This all indicates that it's a variation of bug #1102.
It might be possible to work around the problem by disabling debug info for this particular assembly (we don't currently support disabling debug info for specific assemblies, so I'd have to implement it).
However this is not something we'll be able to implement for 15.3, so I'm bumping the milestone.
I've used changes mentioned in #36 and this unblocked me. Thanks a lot.
But this issue still not solved. I expect meaningful error message at build time, if it is not possible to build correct binaries due to some limitations of a tool in tools chain.
(In reply to Rolf Bjarne Kvinge [MSFT] from comment #37)
> If I enable the linker for all assemblies, the executable size drops to 34MB
> and the crash goes away.
> This may very well be a problem with DocumentFormat.OpenXml.dll, it's an
> impressive assembly with over 64000 methods (~41000 after linking all
> This all indicates that it's a variation of bug #1102.
> It might be possible to work around the problem by disabling debug info for
> this particular assembly (we don't currently support disabling debug info
> for specific assemblies, so I'd have to implement it).
To make things less confusing here, I've filed bug #59634 to track this.
That is great that debug info injection will be optional per each assembly. So problem mentioned in #1102 could be solved.
But for this ticket I would prefer to have meaningful error message that indicates inability to build proper binaries due to 'Section too large' limitation. I.e. I would prefer to stop build process with error message instead of reporting about successful build and further crash at run-time.
Unfortunately detecting this scenario is quite complicated. The proper place to detect this would be in the native linker (ld), which Apple ships. This would involve tracking down the problem inside ld in the first place, come up with a patch, tell Apple about the patch, and hopefully they'd accept it and release it (and it would probably not be released until a year later, with the next major Xcode release). And since it only happens with 32-bit code, and Apple is dropping 32-bit quickly, I'm not even sure they'd care enough to get the patch in.