Bug 29615

Summary: Bug during linking
Product: iOS Reporter: Grigory (Playtika) <GrigoryP>
Component: GeneralAssignee: Zoltan Varga <vargaz>
Status: RESOLVED FIXED    
Severity: normal CC: ddunkin, devender, jatint, jerome.laban, Kent.Green, kumpera, mono-bugs+monotouch, olegi, pj.beaman, prashant.cholachagudd, rolf, tom.philpot, williamd
Priority: Highest    
Version: XI 8.10   
Target Milestone: Untriaged   
Hardware: Macintosh   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:

Description Grigory (Playtika) 2015-04-30 13:36:10 UTC
I have pretty big XML with linker rules and Link All option enabled. NO LLVM.
With SDK Only linker everything works OK.
With LLVM everything works OK.

ld: don't know how to convert instruction ecc9f60c referencing _mono_create_corlib_exception_0.island.3 to thumb in 'SM_Machines_MiniGames_MiniGame270__ConvertResponceToMessagec__Iterator0_Reset' from /Users/grigoryp/Documents/sm.ng/Build/Obj/Debug/SM.IOS/mtouch-cache/SM.Machines.dll.armv7.o for architecture armv7
clang: error: linker command failed with exit code 1 (use -v to see invocation)
MTOUCH: error MT5309: Native linking error: don't know how to convert instruction ecc9f60c referencing _mono_create_corlib_exception_0.island.3 to thumb in 'SM_Machines_MiniGames_MiniGame270__ConvertResponceToMessagec__Iterator0_Reset' from /Users/grigoryp/Documents/sm.ng/Build/Obj/Debug/SM.IOS/mtouch-cache/SM.Machines.dll.armv7.o for architecture armv7
MTOUCH: error MT5201: Native linking failed. Please review the build log and the user flags provided to gcc: -L /Users/grigoryp/Documents/sm.ng/Src/SM/SM.IOS/../../Playtika/Monosyne/Monosyne/ThirdParty/Libs/IOS/unified/ -lFreetype2 -ltremor -lzlib -lBinkiOS -framework AudioToolbox
MTOUCH: error MT5202: Native linking failed. Please review the build log.
Comment 1 Grigory (Playtika) 2015-04-30 13:45:03 UTC
Ah, i also have Debugging enabled. Disabling it fixes the issue.
Comment 2 Rolf Bjarne Kvinge [MSFT] 2015-05-01 03:21:07 UTC
This happens when the native code we generate for one of your assemblies ends up hitting size limitations in the native (Mach-O) file format.

One potential workaround which worked for one client is to add the following to the additional mtouch arguments:

    --gcc_flags -mno-thumb

other workarounds are mentioned here: https://bugzilla.xamarin.com/show_bug.cgi?id=1102#c25 (some of which you've already found).

If you give us access to your entire project (since this is a size issue it's not possible to create a small test case for it) we can have a look and see if we can improve our codegen somewhere.
Comment 8 Grigory (Playtika) 2015-05-06 09:44:21 UTC
Hi, i think it makes sense to look at root of the problem - codegen in Monotouch. I've disassembled biggest obj file (Mach-O one), and found many functions like:

_XXX_System_Runtime_CompilerServices_AsyncMethodBuilderCore_GetCompletionAction_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Threading_Tasks_VoidTaskResult_YYYc__async4_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Threading_Tasks_VoidTaskResult__YYYc__async4_
_XXX_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_AwaitUnsafeOnCompleted_System_Runtime_CompilerServices_TaskAwaiter_1_bool_YYYY__async9_System_Runtime_CompilerServices_TaskAwaiter_1_bool__YYY_
_XXX_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_System_Threading_Tasks_VoidTaskResult_System_Runtime_CompilerServices_IAsyncMethodBuilder_PreBoxInitialization_YYY__async0_YYYY__async0_
_XXX_System_Runtime_CompilerServices_AsyncTaskMethodBuilder_1_YYY_AwaitUnsafeOnCompleted_System_Runtime_CompilerServices_TaskAwaiter_1_YYY_YYY__WaitForScrollc__async3_System_Runtime_CompilerServices_TaskAwaiter_1_YYY__ZZZc__async3_

_XXX_wrapper_delegate_invoke_System_Func_2_object_YYY_invoke_TResult_T_object
and other delegates.

Basically they contain same code. (and for example GetCompetionAction is really 

I believe that i touched problem of huge Mach-O size because of heavy usage of async/await based code.

I see a solution of rewriting code without async/await, but this will contradict with statements from http://xamarin.com/platform about C# features
Comment 9 Rolf Bjarne Kvinge [MSFT] 2015-05-06 09:52:59 UTC
@Zoltan, can you have a look?
Comment 10 Grigory (Playtika) 2015-05-07 03:52:11 UTC
I calculated amount of machine code in one of our assemblies. 
Numbers = percent of total count of A32 instructions per namespace
Small namespaces with <0.5% are omitted 

More that 1/2 of assembly is the code from System namespace or delegates.

0,55 = _SM_Shared_BannerInLobby
3,33 = _SM_Shared_Common
3,14 = _SM_Shared_Gifts
0,67 = _SM_Shared_LevelUp
2,76 = _SM_Shared_LuckySpin
2,41 = _SM_Shared_Machines
4,45 = _SM_Shared_Payments
1,05 = _SM_Shared_PersonalOffers
1,30 = _SM_Shared_PiggyBank
0,62 = _SM_Shared_RegistrationReminders
1,33 = _SM_Shared_Settings
1,31 = _SM_Shared_TotalRewardsSocial
0,59 = _SM_Shared_Tournamania
0,79 = _SM_Shared_Utils
0,85 = _System_Array_InternalArray
1,03 = _System_Array_InternalEnumerator
1,13 = _System_Array_qsort
17,69 = _System_Collections_Generic
1,31 = _System_Linq_Enumerable
0,59 = _System_Reactive_Subjects
19,84 = _System_Runtime_CompilerServices
6,95 = _System_Threading_Tasks
8,79 = _wrapper_delegate_invoke
Comment 11 Grigory (Playtika) 2015-05-07 04:03:51 UTC
To be more clear, what i did:
I took .armv7.llvm.o file, disassembled with IDA, created .map file, calculated sizes of funcs and grouped by func export name substring (till 3rd underscore)
Comment 12 Grigory (Playtika) 2015-05-07 12:48:36 UTC
Copying all code to 1 project reduced binary on 3megs. Seems some System.Collections.Generic code is duplicated for each assembly.

This is an awful solution for a big project we have.
Comment 13 Zoltan Varga 2015-05-07 16:14:06 UTC
These problems usually happen when code uses lots of structs/enums mixed with generics, since the AOT compiler needs to generate different code for each, so
using List<AStruct> and List<BStruct> will cause two list classes to be generated.

I'd suggest converting some of the structs used in the code to classes.

The next major xamarin.ios release contains improvements to this, enums will be handled better, i.e. List<AEnum> and List<BEnum> will be shared.
Comment 14 Grigory (Playtika) 2015-05-07 16:32:52 UTC
Sounds great, as enums by default share same underlying type.
Can we do something with async/await ? 

AsyncMEthodBuilder, VoidTaskResult and some other bcl types are declared as structs. Unfortunately that gives a huge amount of pretty repeating boilerplate(i checked asm - methods differ only by offsets, hopefully it can be parametrized and code can be deduplicated).
Comment 15 Zoltan Varga 2015-05-07 16:38:59 UTC
For a method like:

System_Runtime_CompilerServices_AsyncMethodBuilderCore_GetCompletionAction_System_Runtime_CompilerServices_AsyncVoidMethodBuilder_SM_Machin
es_MiniGames_MiniGame332_Rounds_SaveWendy_WendyRound__OnRestorec__async2_System_Runtime_CompilerServices_AsyncVoidMethodBuilder__SM_Machines_MiniGames_MiniGam
e332_Rounds_SaveWendy_WendyRound__OnRestorec__async2_

Is the SM_Machines_MiniGames_MiniGame332_Rounds_SaveWendy_WendyRound
type a struct type or a normal class ?
Comment 17 Grigory (Playtika) 2015-05-12 09:51:14 UTC
Hi,
Maybe there are some good news with this thing ?

This is an important bug, as all that CompilerServices stuff takes >15% of binary, which is critical on IOS, because Apple has 80 000 000 bytes limitation on 2arch binary.

async/await heavy app can reach this limit pretty fast. 

Of course Xamarin sample apps, chats, RSS readers, calculators would never have this problem. But app i've attached is bigger than that cool presentation stuff, it is an app from market top, that earns big money for years, and is beging rewritten in C#. 
Sad thing is that original app (same functionality) had binary about 30 megs, and we have 90.
Comment 19 Grigory (Playtika) 2015-05-12 10:00:47 UTC
Or if it is by-design - just say that. Then everyone who has encountered same problem will know that this 15% of binary are essential and won't be removed later, and issue should be fixed by another instruments. Partially or fully rewriting into Cpp, or Swift, or whatever. 

But any architectural decisions are quite painful when issue lies on pretty low level (codegen).
Comment 20 Zoltan Varga 2015-05-12 12:03:37 UTC
Most apps don't use that many async methods, probably thats why this issue was not noticed before. It is an implementation problem, c# generates lots of instantiations of the AsyncMethodBuilder etc. classes, and our code generator is not yet good enough to share the code for them.
Comment 21 Zoltan Varga 2015-05-14 12:39:18 UTC
The duplicate code generation for the AsyncMethodBuilder classes should be fixed in mono master 082de058c39a6e6e3b8d6840471bd02ac1ab90d5.
Comment 22 Grigory (Playtika) 2015-05-20 05:58:30 UTC
Great job! Looking forward to try the monotouch build with the fix.
Also wrappers are duplicated, i believe they bring few megs to the build size in code that uses delegates heavily. _XXX_wrapper_delegate_invoke_System_Func_1_System_Collections_Generic_Dictionary_2_string_YYY_invoke_TResult
Comment 24 Grigory (Playtika) 2015-06-03 09:57:18 UTC
Interesting thing is the mono_eh_frame that takes ~15% of binary size. Is there any option for us to change our code in some way to reduce it ?
Comment 25 Zoltan Varga 2015-06-03 14:06:50 UTC
There is no way to reduce that right now, but it shouldn't be that big. Will look at it.
Comment 26 Rodrigo Kumpera 2015-07-08 15:13:31 UTC
The partial fix for async code size will be on the next stable release.
Comment 28 Grigory (Playtika) 2015-08-12 13:50:59 UTC
About binary size:
8.10.4 from Stable produces binary that is 14 megs bigger than 8.9.1.5. We get 89megs so we can't even submit to AppStore. latest 8.11 build are a bit better, but still too bad. They add +6megs to the result of 8.9.1.5.

About this bug: 
We still can't build a debug build, because of this issue. Even when i select ONLY ARMv7 it crashes. Codegen is still inefficient :(

-> REOPEN
Comment 31 Zoltan Varga 2015-08-12 16:00:31 UTC
8.10.4.46 is a bug fix release, the only size related change is the fix for the AsyncMethodBuilder code generation, so it should improve code size related to 8.10.2.
Comment 32 Grigory (Playtika) 2015-09-10 08:42:49 UTC
To be clear:
1. Problem with building in Debug mode still exists. We can only build 1arch in release mode with enabled linker.

2. Binary size problem still exists (we fight for every kilobyte, not to reach 80 megs).

Guys, you should solve it somehow. It is not even funny. 

Are you planning to sell Xamarin only to startup teams that write 3 screen sample apps ?
Comment 34 Zoltan Varga 2015-09-10 13:02:54 UTC
Does the current stable 8.10.4.46 release improve anything for you ? It should only include bug fixes plus the fix for the AsyncMethodBuilder issue, so it should definitely lead to smaller executable sizes than 8.9.1.5.

8.13 is an alpha release so some things might be better, some might be worse.
Comment 35 Jerome Laban 2015-10-07 08:11:58 UTC
I'm having the same issue, related to the binary size. In my case, enabling LLVM lowers the binary size, making it runnable, but not debuggable.
Comment 37 Grigory (Playtika) 2016-01-28 07:49:11 UTC
X.I. 9.4 helps with that.
So workaround is to update (but keep in mind that 9.4 is very buggy, exceptions are not caught in "catch"es in some cases)
Comment 38 Zoltan Varga 2016-01-28 18:34:15 UTC
That bug is probably:
https://bugzilla.xamarin.com/show_bug.cgi?id=37273
Comment 39 Prashant [MSFT] 2017-07-26 11:19:40 UTC
@Grigoty @Zoltan,

I don't see any activity on this bug in 18months! Commonent #37 suggested that updating to latest version did fix the issue, so marking this as resolved. Please reopen the bug if you still encounter this issue