Bug 42395

Summary: Build runs indefinitely and never finishes
Product: iOS Reporter: Adam Hartley [MSFT] <adam.hartley>
Component: Mono runtime / AOT compilerAssignee: Zoltan Varga <vargaz>
Status: VERIFIED FIXED    
Severity: normal CC: avidanc, brendan.zagaeski, chrisntr, david, d.lilley, dominic, jon.goldberger, j.pasque, kumpera, kyle.white, manuel.delapena, matteo, mohitk, mono-bugs+monotouch, mruston, reinis, rolf, sebastien
Priority: ---    
Version: XI 9.8 (tvOS / C7)   
Target Milestone: 10.0.1 (C8SR0)   
Hardware: PC   
OS: Mac OS   
See Also: https://bugzilla.xamarin.com/show_bug.cgi?id=41807
Tags: Is this bug a regression?: Yes
Last known good build:
Attachments: Build Logs for LLVM ON (Never finishes) and LLVM OFF (Successful)

Description Adam Hartley [MSFT] 2016-07-07 08:20:14 UTC
When building in the Release or App Store configuration, the build runs indefinitely and never finishes.

The customer rolled back to Cycle 6 and the same builds are completing fine.

Logs, version info and sample will be in the following (private) comment.
Comment 2 matteo 2016-07-07 08:37:31 UTC
*** Bug 42343 has been marked as a duplicate of this bug. ***
Comment 3 Sebastien Pouliot 2016-07-08 19:37:56 UTC
@Adam have you been able to duplicate this ?

My build fails as several components are missing from the linked sample.
Comment 4 Adam Hartley [MSFT] 2016-07-08 21:01:56 UTC
Wasn't able to check the sample before filing as I was having issues with my Mac (now reinstalled).

For the NGrid and NChart components you'll need to be logged in for them to be downloaded from the store automatically. Looks like the other "ComponentPro" packages can be installed via this component https://components.xamarin.com/view/UltimateFtp

Next add references to the existing projects PL.Dati.iOS and PL.Dati.OutputDati.iOS - the latter you will have to do a manual nuget update and accept the license for MoonSharp.
Comment 5 Julian Pasqué 2016-07-13 15:04:55 UTC
I have the same problem. In Release-mode my build runs for ever.
Even rolling back to Cycle 6 didn't fix my problem.

That is verry timeconsuming
Comment 6 Rolf Bjarne Kvinge [MSFT] 2016-07-15 10:00:45 UTC
@Julian, we'll need a way to reproduce this in order to fix it, so if you can share your project with us (or a test project), we'll have a look asap.
Comment 7 Julian Pasqué 2016-07-15 10:04:11 UTC
Currently it is working again. I switched to the current Beta-Channel.
Comment 8 Reinis 2016-07-15 18:22:51 UTC
I am having the same problem.
These are processes that have hanged - 
8056 ??         0:42.69 /Library/Frameworks/Xamarin.iOS.framework/Versions/Current/bin/mtouch --cache /Users/reinis/Projects/DefaultCollection/_git/SonarworksNew/MobileApp/ReferenceCloud/iOS/obj/iPhone/Release/mtouch-cache --dev /Users/reinis/Projects/DefaultCollection/_git
 8370 ??         0:11.01 /Library/Frameworks/Xamarin.iOS.framework/Versions/9.8.1.4/bin/arm-darwin-mono-sgen --debug --llvm -O=gsharedvt -O=float32 --aot=mtriple=armv7-ios,data-outfile=/Users/reinis/Projects/DefaultCollection/_git/SonarworksNew/MobileApp/ReferenceCloud/iOS/o
 8479 ??        14:38.22 /Library/Frameworks/Xamarin.iOS.framework/Versions/9.8.1.4/LLVM/bin/llc -mattr=+vfp2,-neon,+d16 -asm-verbose=false -mtriple=armv7-ios -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_ReferenceCloud_eh_frame -relocation-mode


Sometimes I am able to get it to build Release versions but have not figured out what triggers success.

I am able to build Debug versions and there are no problems when building Release for simulator.

It only fails when building Release for device. I think this started after upgrade to latest versions and upgrade of some packages (SQLite, Xamarin.Forms)
Comment 9 David Lilley 2016-07-22 10:30:28 UTC
I also had the same issue.
I upgraded to Beta and this did not work.
I rolled back to cycle 6 and also discovered new compiler constants have been added to project for DEBUG but not Release. I deleted these and it all works in cycle 6. 

__UNIFIED__;__MOBILE__;__IOS__

I will check later if adding them to Release in cycle 7. but first need to ship an App.
Comment 10 David Lilley 2016-07-28 11:45:01 UTC
I have recheck in cycle 7 ensuring these __UNIFIED__;__MOBILE__;__IOS__ are in the release build and it made no difference.

I am using Xamarin Studio for Mac.

A Colleague of mine suggested to use single threaded build ( an option of visual studio  not xamarin mac) as he had issues of multithreaded building with shared projects. Unsure if that is the problem ?

=== Xamarin Studio Community ===

Version 6.0.1 (build 9)
Installation UUID: 0ffe13e5-2d69-479c-9d2a-277e6808626d
Runtime:
	Mono 4.4.1 (mono-4.4.0-branch-c7sr0/4747417) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 404010000

=== Xamarin.Profiler ===

Not Installed

=== Apple Developer Tools ===

Xcode 7.3.1 (10188.1)
Build 7D1014

=== Xamarin.iOS ===

Version: 9.8.1.4 (Xamarin Studio Community)
Hash: 3cf8aae
Branch: c7sr0
Build date: 2016-06-20 16:09:58-0400

=== Xamarin.Android ===

Not Installed

=== Xamarin Android Player ===

Not Installed

=== Xamarin.Mac ===

Version: 2.8.1.4 (Xamarin Studio Community)

=== Build Information ===

Release ID: 600010009
Git revision: e879ce52093257c5c386ad7e390dfaa937fa7f90
Build date: 2016-06-24 11:53:43-04
Xamarin addins: a9252e6df4851fbbed1f9c6228e7b6dd1b475ac5
Build lane: monodevelop-lion-cycle7-sr0

=== Operating System ===

Mac OS X 10.11.5
Darwin DlilleyMBP.local 15.5.0 Darwin Kernel Version 15.5.0
    Tue Apr 19 18:36:36 PDT 2016
    root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
Comment 11 Manuel de la Peña 2016-08-04 09:33:24 UTC
Hello,

As @rolf mentioned, we need a way to reproduce this. Can you please provide and project that does have the issue?
Comment 12 David Lilley 2016-08-04 09:46:34 UTC
Hello,

Do you have any clues to what the problem may be ? We not happy to send entire solution  over but we could try and make a small sample for you based upon our solution?

(FYI we are not using any nuggets etc)
Comment 13 matteo 2016-08-04 09:54:11 UTC
I've already send my entire solution
Comment 14 David Lilley 2016-08-18 07:32:46 UTC
Thanks Matteo,

Xamarin do you need any more info ?
Comment 15 Sebastien Pouliot 2016-08-18 12:26:11 UTC
No test case to reproduce this as reached us. This is needed along with all the versions of the software being used.

The easiest way to get exact version information is to use the 
"Xamarin Studio" menu, "About Xamarin Studio" item, "Show Details" 
button and copy/paste the version informations (you can use the 
"Copy Information" button).
Comment 16 matteo 2016-08-18 12:38:02 UTC
I send my solution to Adam Hartley via Dropbox. Here's system info

=== Xamarin Studio Business ===

Version 6.0.1 (build 9)
Installation UUID: 00b370de-707e-48ed-b366-181d03bf149e
Runtime:
    Mono 4.4.1 (mono-4.4.0-branch-c7sr0/4747417) (64-bit)
    GTK+ 2.24.23 (Raleigh theme)

    Package version: 404010000

=== Xamarin.Profiler ===

Version: 0.33.2
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Apple Developer Tools ===

Xcode 7.3.1 (10188.1)
Build 7D1014

=== Xamarin.Mac ===

Not Installed

=== Xamarin.iOS ===

Version: 9.6.2.4 (Xamarin Business)
Hash: d8bedd0
Branch: master
Build date: 2016-05-05 17:43:01-0400

=== Xamarin.Android ===

Version: 6.1.1.1 (Xamarin Business)
Android SDK: /Users/matteo/Library/Developer/Xamarin/android-sdk-mac_x86
    Supported Android versions:
        2.3 (API level 10)
        4.4 (API level 19)

SDK Tools Version: 24.0.2
SDK Platform Tools Version: 1602
SDK Build Tools Version: 19.1

Java SDK: /usr
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-468-11M4833)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-468, mixed mode)

Android Designer EPL code available here:
https://github.com/xamarin/AndroidDesigner.EPL

=== Xamarin Android Player ===

Not Installed

=== Build Information ===

Release ID: 600010009
Git revision: e879ce52093257c5c386ad7e390dfaa937fa7f90
Build date: 2016-06-24 11:53:43-04
Xamarin addins: a9252e6df4851fbbed1f9c6228e7b6dd1b475ac5
Build lane: monodevelop-lion-cycle7-sr0

=== Operating System ===

Mac OS X 10.11.5
Darwin MacBook-Pro-di-Matteo.local 15.5.0 Darwin Kernel Version 15.5.0
    Tue Apr 19 18:36:36 PDT 2016
    root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64
Comment 17 Adam Hartley [MSFT] 2016-08-19 14:59:56 UTC
@Matteo Sample, version info and verbose log where attached privately when the bug was opened
Comment 19 matteo 2016-08-22 06:51:19 UTC
@Adam Do I have to do something else?
Comment 20 David Lilley 2016-08-22 10:30:42 UTC
I have done some testing today, checking out what the difference between Release and Debug and discovered if i switch off "LLVM Optimizing Complier"  I can get my solution build in cycle 7, (In Cycle 6 it builds fine with LLVM on.)

Attached my detailed logs for LLVM on and off.
Comment 21 David Lilley 2016-08-22 10:34:22 UTC
Created attachment 17153 [details]
Build Logs for LLVM ON (Never finishes) and LLVM OFF (Successful)
Comment 22 Avidan Chen 2016-08-24 08:02:54 UTC
I can confirm this issue happens for us as well.This forces us to remain on Cycle 6 in our CI machine until this issue is resolved. Please fix it as soon as possible.
Comment 23 Jon Goldberger [MSFT] 2016-08-25 22:28:12 UTC
I have another customer who may be hitting this bug. Same symptoms, LLVM on - build hangs, LLVM off - Build works. But it also works with LLVM on and Thumb-2 instruction set enabled. I am getting permission to get the sample project to add to this report. Will pst as soon as I can. The sample project is a simple iOS Snngle view app that references the XFinium library [1] and the only code added to the app project is just instantiating a type from the XFinium library. 

[1] http://www.xfiniumpdf.com/xfinium-pdf-crossplatform/xfinium-pdf-overview.html
Comment 24 Jon Goldberger [MSFT] 2016-08-25 22:32:55 UTC
Here is a build log with -v -v -v -v generated from the sample project mentioned in comment 23 above: https://gist.github.com/jgold6/85d07adf0335b37b0315e37d3e96efa6

This was collected after a few minutes when I cancelled the build. NOte that until I cancelled the build, there was no output shown in the Build Output window at all. 

I also noted several llc processes in Activity Monitor all using massive amounts of cpu time, and these processes did not stop when Xamarin Studio was quit. When using the Thumb-2 option, the build succeeded and the llc processes came and went and stopped when the build finished.
Comment 27 Rolf Bjarne Kvinge [MSFT] 2016-08-26 07:11:56 UTC
(In reply to Jon Goldberger from comment #24)
> Note that until I cancelled the build, there was no output shown in the Build Output window at all. 

This is a change in how Xamarin Studio shows the output (I've asked about it and it's by design).
Comment 28 Rolf Bjarne Kvinge [MSFT] 2016-09-08 15:49:48 UTC
I can reproduce with the same from comment #24, by building the release configuration for device.

It seems llc runs into some sort of infinite loop with Xfinium.Pdf.Xamarin.iOS.dll:

    > ps aux|grep llc
rolf             69405  99.8  1.6  2782840 262060   ??  R     5:45PM   1:06.62 /Library/Frameworks/Xamarin.iOS.framework/Versions/git/LLVM/bin/llc -mattr=+vfp2,-neon,+d16 -asm-verbose=false -mtriple=armv7-ios -disable-gnu-eh-frame -enable-mono-eh-frame -mono-eh-frame-symbol=_mono_aot_Xfinium_Pdf_Xamarin_iOS_eh_frame -relocation-model=pic -filetype=obj -o /Users/rolf/Downloads/HTPdfTest/HTPdfTest/obj/iPhone/Release/mtouch-cache/Xfinium.Pdf.Xamarin.iOS.dll.armv7-llvm.o /Users/rolf/Downloads/HTPdfTest/HTPdfTest/obj/iPhone/Release/mtouch-cache/Xfinium.Pdf.Xamarin.iOS.dll.armv7.s.opt.bc
Comment 30 Zoltan Varga 2016-09-09 00:16:05 UTC
Can repro this.
Comment 31 Zoltan Varga 2016-09-09 01:15:17 UTC
Should be fixed by mono master 2acdc49ff6b86bcd6179aa5bd6d861d37709dcf7.
Comment 32 Rolf Bjarne Kvinge [MSFT] 2016-09-09 06:06:04 UTC
@Zoltan, this looks like something that's easy to backport to mono 4.6, is that right?

Also the fix works for me.

xamarin-macios PR (master): https://github.com/xamarin/xamarin-macios/pull/810
Comment 33 Zoltan Varga 2016-09-09 12:44:30 UTC
Yes, its ok to backport.
Comment 34 Rolf Bjarne Kvinge [MSFT] 2016-09-09 12:45:27 UTC
Fixed (master): https://github.com/xamarin/xamarin-macios/commit/7a2baeac44ddfefd68f898948079e4c6aa486e5d
Comment 36 Mohit Kheterpal 2016-09-13 15:21:13 UTC
I am able to reproduce this issue with the sample project given in comment 25 with Cycle 8 beta builds and also observed that this issue has been fixed with latest build of master xamarin.ios-10.1.0.8_468b3a39cc21a8816489cbc90c977d7b39f7187a

Screencast : http://www.screencast.com/t/KXVbBvV4QD

Hence closing this issue by marking it as Verified.

thanks
Comment 37 Avidan Chen 2016-09-14 08:41:45 UTC
I have installed the latest Cycle 8 build (stable) and this issue still occurs. When are we supposed to get the fix?
Comment 38 Rolf Bjarne Kvinge [MSFT] 2016-09-14 10:12:06 UTC
@Avidan, we're _trying_ to get it into the first service release for Cycle 8 (the plan is to release that in a week or so).
Comment 39 Avidan Chen 2016-09-14 10:27:55 UTC
@Rolf Bjarne Kvinge - Thank you!
Comment 40 Rolf Bjarne Kvinge [MSFT] 2016-09-15 05:08:12 UTC
@Avidan, can you try disabling LLVM in the project's iOS Build options for those configurations and see if that works around the issue?
Comment 41 Avidan Chen 2016-09-15 05:41:07 UTC
@Rolf - I tried it and it does work around the issue. However I'm a bit reluctant to use it for production apps as I'm not sure how big the performance hit is going to be. We need to upload a production build next week... Hopefully we'll have the service release soon. 

Thanks!
Comment 42 Rolf Bjarne Kvinge [MSFT] 2016-09-15 05:46:40 UTC
@Avidan, unless you're doing a lot of math in managed code (image processing for instance, or your app is a game with much logic written in C#), the difference should be minimal to non-existent. If you never saw performance problems in debug mode (where LLVM is disabled), you won't have any problems disabling LLVM for release mode either.
Comment 43 Mohit Kheterpal 2016-09-19 12:15:11 UTC
I have checked this issue and observed that fix for this issue has been merged in C8SR0 as shown in screencast : http://www.screencast.com/t/96vTzqT8

Using xamarin.ios-10.0.1.5

Thanks