Bug 53602 - Extremely slow iOS build with a few projects references
Summary: Extremely slow iOS build with a few projects references
Status: RESOLVED DUPLICATE of bug 54467
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: iOS ()
Version: 4.3.0 (C9)
Hardware: PC Windows
: Normal normal
Target Milestone: 15.2.2
Assignee: Bugzilla
Depends on:
Reported: 2017-03-21 06:35 UTC by Virgile Bello
Modified: 2017-06-21 19:41 UTC (History)
8 users (show)

Is this bug a regression?: ---
Last known good build:

Repro project (23.08 KB, application/x-zip-compressed)
2017-03-21 06:48 UTC, Virgile Bello

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Virgile Bello 2017-03-21 06:35:04 UTC
Have a bunch of iOS library projects (with project references).

Due to how Xamarin target files are built, each project recursively call MSBuild on subprojects, resulting in exponential build times as you add a few projects.

With just a few projects, I can easily reach *several minutes* build, and millions of line of log in the MSBuild diagnostic log.

I could narrow it down to `GetBuiltProjectOutputRecursive` target, which seems to recursively call MSBuild at each level.
2>    32732 ms  GetBuiltProjectOutputRecursive            24 calls
Removing that target make everything as fast as when I build project on other platforms.

Note: MS targets try to totally avoid this kind of case (cf GetCopyToOutputDirectoryItems documentation) as it quickly explodes in terms of complexity.

Could it be made non recursive or rewritten in a better way?
Comment 1 Virgile Bello 2017-03-21 06:48:54 UTC
Created attachment 20484 [details]
Repro project

Added a repro project with 8 class libraries (with references)
Try to build ClassLibrary8:
4>    61961 ms  MSBuild                                  131 calls

4>     1028 ms  c:\users\virgile.sskk\documents\visual studio 2015\Projects\SlowBuild\ClassLibrary1\ClassLibrary1.csproj  67 calls
4>                  0 ms  GetTargetPath                              1 calls
4>                  0 ms  GetNativeManifest                          1 calls
4>                  0 ms  GetCopyToOutputDirectoryItems              1 calls
4>               1027 ms  GetBuiltProjectOutputRecursive            64 calls

We can see that the most inner project is called 67 times (and would grow even higher if I add one more level)
Comment 2 Virgile Bello 2017-03-21 06:56:02 UTC
Note: the initiator seems to be _ConvertDebuggingFiles => _CollectAssemblies => GetBuiltProjectOutputRecursive
Comment 3 Manuel de la Peña [MSFT] 2017-03-21 16:52:56 UTC

I have tested you project (my environment is printed bellow) and did not notice a very slow build (gist: https://gist.github.com/mandel-macaque/71c720c2aacb134f9a39a25733476041) as you can see  from the logs:

Build started 3/21/2017 5:42:45 PM.
Time Elapsed 00:00:05.4026510

Can you please provide the environment you are using? My environment is as follows:

=== Xamarin Studio Community ===

Version 6.4 (build 5)
Installation UUID: 01060673-5bee-4cf4-a4c2-5e36a18d39a2
	Mono 4.8.0 (mono-4.8.0-branch/8f6d0f6) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 408000520

=== NuGet ===


=== Xamarin.Profiler ===

Version: 1.3.3
Location: /Applications/Xamarin Profiler.app/Contents/MacOS/Xamarin Profiler

=== Apple Developer Tools ===

Xcode 8.2.1 (11766.1)
Build 8C1002

=== Xamarin.iOS ===

Version: (Xamarin Studio Community)
Hash: e884b914
Branch: CFNetwork-lost-connection
Build date: 2017-03-21 11:07:24+0100

=== Xamarin.Mac ===

Version: (Xamarin Studio Community)

=== Xamarin.Android ===

Version: (Xamarin Studio Community)
Android SDK: /Users/mandel/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		4.0.3 (API level 15)
		4.3   (API level 18)
		4.4   (API level 19)
		5.0   (API level 21)
		6.0   (API level 23)

SDK Tools Version: 25.2.3
SDK Platform Tools Version: 25.0.1
SDK Build Tools Version: 25.0.1

Java SDK: /usr
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Android Designer EPL code available here:

=== Xamarin Inspector ===

Version: 1.2.0-rc.2
Hash: 15b00be
Branch: d15-1
Build date: Mon, 06 Mar 2017 15:58:04 GMT

=== Build Information ===

Release ID: 604000005
Git revision: a04f5191c0d91121215fbbaa866f9d420521b6ee
Build date: 2017-03-02 17:05:20-05
Xamarin addins: 5f8e7287f81b61afea39453a5f2228642fd34e52
Build lane: monodevelop-lion-master

=== Operating System ===

Mac OS X 10.12.3
Darwin MacBook-Pro.local 16.4.0 Darwin Kernel Version 16.4.0
    Thu Dec 22 22:53:21 PST 2016
    root:xnu-3789.41.3~3/RELEASE_X86_64 x86_64

=== Enabled user installed addins ===

Addin Maker 1.3.2
StyleCop Support
Comment 4 Virgile Bello 2017-03-22 09:01:44 UTC
Did you try to enable the diagnostic MSBuild log?

Lot of umnecessary work is going on and it gets worse the more projects you have.
Comment 5 Virgile Bello 2017-03-22 17:43:13 UTC
Note: this is a regression, this problem started to all computr that I have upgraded to latest stable.

Also, on our TeamCity server, iOS build of our product takes 8 min and produce a 12mb log (filled with the same project being processed over again and again) whereas all other platforms (including Xamarin.Android) take ~40 seconds for a 1mb log.
Comment 6 Jeffrey Stedfast 2017-03-22 20:07:49 UTC
A regression compared to what version?

Also: none of these targets exist in the Mac Xamarin.iOS.Common.targets:


Is this in the Windows-side of the MSBuild targets?

If so, then this probably needs to be reassigned back to the VS team.
Comment 7 Virgile Bello 2017-03-23 02:30:46 UTC
Yes, it is Windows-side from Visual Studio.
Comment 8 Virgile Bello 2017-03-23 02:37:14 UTC
Additional details: the targets mentionned before are in
C:\Program Files (x86)\MSBuild\14.0\Microsoft.Common.Targets\ImportAfter\Xamarin.Common.targets

Version that *have* the issue:

Version that *doesn't have* the issue:

So the regression is very recent!
Comment 9 Jeffrey Stedfast 2017-03-23 15:37:57 UTC
Thanks! This sounds like the problem is on the Visual Studio side.
Comment 10 Daniel R. Regner 2017-04-07 20:11:35 UTC
I saw similar behavior (deep recursion during builds), and the work-around to edit the GetBuiltProjectOutputRecursive MSBuild targets in the discussion at bug #54467 worked for me to drop build times by an order of magnitude.
Comment 11 Virgile Bello 2017-04-19 09:16:30 UTC
Recently our team is having the same problem with Android as well (target _SetLatestTargetFrameworkVersion and GetBuiltProjectOutputRecursive).

This is getting a *huge blocker* for us! (esp. since VS2017 we can't even install older working version)
Comment 12 Virgile Bello 2017-04-19 09:32:02 UTC
Quick update: https://bugzilla.xamarin.com/show_bug.cgi?id=54467 seems to fix it for us as well (thanks Daniel for the tip).

Any chance this gets released in 15.1?
Comment 13 Adrian Alonso 2017-06-21 19:41:11 UTC
Hi Virgile, the fix has been already released as part of 15.2.3. Thanks

*** This bug has been marked as a duplicate of bug 54467 ***