Bug 52160 - Building is extremely slow
Summary: Building is extremely slow
Status: RESOLVED DUPLICATE of bug 47288
Alias: None
Product: Android
Classification: Xamarin
Component: General ()
Version: 7.0 (C8)
Hardware: Macintosh Mac OS
: --- normal
Target Milestone: ---
Assignee: dean.ellis
Depends on:
Reported: 2017-01-31 20:45 UTC by Joey Z
Modified: 2017-02-07 21:16 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

Attachment: example "good" diagnostic incremental build log for Acquaint sample on 2012 MacBook Air (88.70 KB, application/zip)
2017-02-02 02:48 UTC, Brendan Zagaeski (Xamarin Team, assistant)

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Joey Z 2017-01-31 20:45:19 UTC
Building my application is extremely slow.  We have a team of 4 people.  Others say that building takes them about 10 minutes maximum.  For me, it's taking about 15-20 minutes.  I'm having a lot of issues daily that are only resolvable by doing a build-clean and/or clearing the bin & obj directories.  But when I do that, it literally takes about 20 minutes to build the project.  This has resulted in a dramatic reduction in productivity.  Can you please help me?  

My mac is a 2.5GHZ i7 with 16GB of memory.  

Xamarin Studio Community
Version 6.1.5 (build 0)
Installation UUID: dc3ef2f5-9782-491b-8b9e-be9898e8bc76
	Mono 4.6.2 (mono-4.6.0-branch/ac9e222) (64-bit)
	GTK+ 2.24.23 (Raleigh theme)

	Package version: 406020016


Not Installed

Version: (Xamarin Studio Community)
Android SDK: /Users/joeyz/Library/Developer/Xamarin/android-sdk-macosx
	Supported Android versions:
		5.0 (API level 21)
		5.1 (API level 22)
		6.0 (API level 23)
		7.0 (API level 24)

SDK Tools Version: 25.2.5
SDK Platform Tools Version: 25.0.3
SDK Build Tools Version: 25.0.2

Java SDK: /usr
java version "1.8.0_101"
Java(TM) SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

Android Designer EPL code available here:

Xamarin Android Player
Version: 0.6.5
Location: /Applications/Xamarin Android Player.app

Apple Developer Tools
Xcode 8.2.1 (11766.1)
Build 8C1002

Version: (Xamarin Studio Community)
Hash: 7beaef4
Branch: cycle8-xi
Build date: 2016-12-20 02:58:14-0500

Version: (Xamarin Studio Community)

Xamarin Inspector
Hash: 1f3067d
Branch: master
Build date: 11/15/2016 1:13:59 PM

Build Information
Release ID: 601050000
Git revision: 7494718e127af9eaec45a3bd6282d3da927488bd
Build date: 2017-01-17 10:31:01-05
Xamarin addins: c92d0626d347aaa02839689eaac2961d24c9f446
Build lane: monodevelop-lion-cycle8

Operating System
Mac OS X 10.11.6
Darwin Pako-2.local 15.6.0 Darwin Kernel Version 15.6.0
    Mon Jan  9 23:07:29 PST 2017
    root:xnu-3248. x86_64

Enabled user installed addins
Xamarin Inspector
Comment 1 Joey Z 2017-01-31 21:05:42 UTC
I timed a build, with no other applications running on my computer, only Xamarin, I began building at 12:44 and it finished building at 12:56.  

Here's the output from the Errors Build Output tab.  But I think it's only logging warnings.  Is there a way I can turn on full diagnostic logging?  

Here is the link to the output
Comment 2 Joey Z 2017-01-31 22:23:58 UTC
Also note that our solution has a lot of image assets associated with it.  I did read about this: https://bugzilla.xamarin.com/show_bug.cgi?id=40726  But we are developing on macs so there shouldn't be any network image transfers.
Comment 3 Jonathan Pryor 2017-01-31 22:41:46 UTC
@Joey: You can enable diagnostic build output by following these instructions:


> I'm having a lot of issues daily that are only resolvable by doing
> a build-clean and/or clearing the bin & obj directories.

This sounds like the *actual* problem. Building from a clean state can be expected to take awhile, as most of what we do to decrease build times relies on caching build artifacts, and if you nuke the bin & obj directories you're removing all of the cached build artifacts.

Could you elaborate on the issues you're running into which necessitate clearing the bin & obj directories?
Comment 4 Joey Z 2017-01-31 22:58:43 UTC
Many random daily issues. To many to count. Sometimes it's a build failure. Sometimes I'll be modifying code, and my changes won't seem to be actually getting applied to the app once it's built. I cannot why, but a build clean + reset the simulator is the only thing that works. Often times when Xamarin has an update, after I update I get error messages and cannot build, so I'm required to clean + reset the simulator.
Comment 5 Joey Z 2017-02-01 18:57:07 UTC
Another issue, last night Xamarin was building just fine.  Suddenly this morning I'm getting "execution failed" so i'm forced to build clean.
Comment 6 Joey Z 2017-02-01 21:24:47 UTC
So i'm building directly to device only now (vs simulator) since my I haven't been able to resolve this "execution failed" issue. Turns out a build clean doesn't fix that... But if I successfully build the app, then change 1 single file and save it.  Then build again, how long should that take to build?  Because it's taking 5 minutes.  And that was changing only 1 file in 1 project.  Is there anything I can do to speed this up?  My productivity has dropped dramatically due to this.
Comment 7 Brendan Zagaeski (Xamarin Team, assistant) 2017-02-02 02:48:55 UTC
Created attachment 19689 [details]
Attachment: example "good" diagnostic incremental build log for Acquaint sample on 2012 MacBook Air

I think Comment 4 - Comment 6 might be heading away a little bit from where we'd like to be ideally on a bug report: 1 "minimal, complete, and verifiable" problem (to borrow some vocabulary from Stack Overflow's guides [1]).

[1] https://stackoverflow.com/help/mcve

I would suggest that for the "one problem" for this bug report (Bug 52160), we focus on the 5 minute build time for the project after you make 1 change in 1 file.

## Next steps

There are at least a couple possible approaches that could be useful as initial steps in investigating this problem.

1. Establish a baseline for the Android build performance on your system using some known sample and compare that to the build performance on another system.

For example, I have provided some numbers below for the current version of the Acquaint prebuilt sample (https://www.xamarin.com/prebuilt) app:

(a) Download and unzip the source code.

(b) Open the App/Acquaint.Native.sln in Xamarin Studio.

(c) Allow Xamarin Studio to restore the packages, and build the "Acquaint.Native.Droid" project once in the debug configuration.

(d) Close Xamarin Studio (we won't need it for the remaining steps).

(e) In a Terminal.app command prompt run the following command to get the "no change" build time:

time xbuild /v:diagnostic /t:Build /p:Configuration=Debug App/Acquaint.Native/Acquaint.Native.Droid/Acquaint.Native.Droid.csproj > /dev/null

### 2 sample results from a 2012 MacBook Air (that was otherwise mostly idle)
> real	0m5.762s
> real	0m5.738s

(f) In the Terminal.app command prompt run the following command to update the time stamp on one file:

touch App/Acquaint.Native/Acquaint.Native.Droid/MainApplication.cs

(g) Now re-run the xbuild command and log the diagnostic build output:

time xbuild /v:diagnostic /t:Build /p:Configuration=Debug App/Acquaint.Native/Acquaint.Native.Droid/Acquaint.Native.Droid.csproj > "$HOME/Desktop/AcquaintIncrementalBuild.txt"

### 2 sample results from a 2012 MacBook Air
> real	0m6.768s
> real	0m6.762s
(The sample diagnostic build output for the second run is attached.)

(h) Finally, collect the rebuild time for the full-length build time.

time xbuild /v:diagnostic /t:Rebuild /p:Configuration=Debug App/Acquaint.Native/Acquaint.Native.Droid/Acquaint.Native.Droid.csproj > /dev/null

### 2 sample results from a 2012 MacBook Air
> real	0m36.233s
> real	0m36.795s

(i) Report back with the timings from steps (e), (g), and (h), and zip up and attach back the diagnostic build output from your Desktop from step (g) (following the example in this comment).

2. If the Acquaint sample does not show an obvious discrepancy (longer build times) on your system compared to the sample numbers from the MacBook Air, then one possible alternate way forward is to carefully study the diagnostic build output from your full original project at the equivalent of step (g) to look for MSBuild Targets that are being re-run when their corresponding Inputs have not in fact changed [2].  I would in fact recommend attaching back that diagnostic build output for your full project in addition to the results from part 1 no matter what results you get for part 1.

[2] (For reference, see also "Incremental Builds" https://msdn.microsoft.com/en-us/library/ee264087.aspx )

Another possible approach could be if you are able to share your full project or a trimmed down version of it that still demonstrates the problem on your machine.  (You can optionally share that project in a new bug report that you mark "Xamarin Team Only" so it will only be visible to the Xamarin team.)

Thanks in advance!

## Testing environment info (brief)

Mono 4.6.2 (mono-4.6.0-branch/ac9e222)

Mac OS 10.11.6
Comment 8 Joey Z 2017-02-02 18:41:39 UTC
Hi Brendan,

Thanks for taking the time to help me with such detailed instructions!  So I downloaded that sample and it doesn't build right out of the box.  It's flagging with an error of 

target named 'GetTargetPath' not found in the project

It seems people are suggesting that, in order to fix this I need to modify the csproj.  I'm attempting to resolve it right now.  If you have a suggestion feel free to toss it my way. 

** Update **

I resolved the issue.  I had to manually modify the csproj and add in the Outputs="$(TargetPath)"   to the <Target>.  It seems the setup that built this sample wasn't out of the box compatible with my xamarin studio setup, which is the latest stable version of everything :-(  I'll continue now
Comment 9 Joey Z 2017-02-02 19:07:36 UTC
(e)  No other applications running (aside from Chrome browser and terminal)
real	0m6.563s
user	0m3.275s
sys	0m0.798s

(f) done

real	0m7.601s
user	0m4.183s
sys	0m0.950s

Here is the file it populated https://drive.google.com/open?id=0B5x7JEZ8aQihTUppUUV5YnBSS0E

real	0m24.427s
user	0m51.902s
sys	0m3.776s

Moving on to step 2 now.
Comment 10 Brendan Zagaeski (Xamarin Team, assistant) 2017-02-02 20:08:25 UTC
## Assignment update

Assigning this bug to myself temporarily while I'm helping examine the results from Comment 8 and Comment 9.  (But Xamarin.Android team of course feel free to take it back if you catch something I miss that looks like the key to the story.)
Comment 11 Brendan Zagaeski (Xamarin Team, assistant) 2017-02-02 21:02:05 UTC
> target named 'GetTargetPath' not found in the project

> It seems the setup that built this sample wasn't out of the box
> compatible with my xamarin studio setup, which is the latest stable
> version of everything

Hmm.  It might be worth taking a gamble that this is an important symptom of a larger problem.   I specifically followed only the exact steps I listed in Comment 7, and I was using the current stable versions too (as noted at the bottom of my comment, and as matches your versions from Comment 0).  As an extra confirmation, I just deleted the unzipped Acquaint folder and re-tried steps (a - e), and again I hit no errors.  The fact that you are getting an error could then already indicate something out-of-place in your environment.  Based on your earlier description of "many random daily issues," I am tempted to recommend that we fork discussion of this symptom into a new Stack Overflow question for follow-up, assuming that you can reliably replicate it by repeating steps (a - c) (or maybe a - e?) from Comment 7.

As one initial clue, I believe the 'GetTargetPath' Target should be defined in the "Microsoft.Bcl.Build.targets", which is part of one of the NuGet packages and is included into the "Acquaint.Native.Droid.csproj" file build via an `<Import Project=` line:


If `<Import Project=` elements aren't working correctly for you, that is quite possibly an interesting thing for us to follow-up on.

### Next steps

If you like, if you can confirm the precise steps you follow to replicate that "'GetTargetPath' not found error", I'll draft up and attach back a template Stack Overflow question that you can post, and then I'll follow-up with you on that Stack Overflow question.

## Back to our "one symptom" for this bug report (the longer than expected incremental build times)

The results from step 2 will indeed potentially be interesting.  I'm already somewhat surprised that your build times for (e) and (g) were longer, especially when the full build was indeed shorter than mine.  The 2012 MacBook Air I was testing on is only a 2.0 GHz 3rd generation Core i7 (I7-3667U), so I'd imagine your processor beats this one.

One factor I forgot to mention earlier is that I unzipped the project to the main drive of the MacBook (the internal soldered-in solid-state storage).  I would assume you did the same, but if not, then be sure to mention what kind of drive you are using so we have that bit of information recorded here too.

### Next steps

I will take a little bit of time to look through the diagnostic build output you linked and compare it to the output I attached in Comment 7 in case anything jumps out as obviously problematic.
Comment 12 Joey Z 2017-02-02 21:50:23 UTC
So is a doc with results from step 2.)  https://drive.google.com/open?id=1jYJMs_uUm4aum3gPIq4pxj5KDrWtUtv5bgXZnrG-gpY

And here's the txt file with the diagnostic output  https://drive.google.com/open?id=0B5x7JEZ8aQihMVFBYW03d3BleW8
Comment 13 Joey Z 2017-02-02 22:04:56 UTC
So I tried reproducing the "GetTargetPath" issue again, and no luck.  This is strange, I did the same thing and it's working.  I checked the csproj file and it does not have the Outputs="$(TargetPath)" in the Target yet it still works.  And that was the clear solution earlier.  To make things funnier, now I can build to the simulator all the sudden!  Before I was getting that "execution failed", but now i'm not...  Let's not think about that right now as we're already getting spread thin.  

So I even went back to the first app-acquaint folder that I unzipped, tried it, and it still works.  So I removed that Outputs="$(TargetPath)" line from it's csproj and it WORKS!  I'm just confused now. I'm going to reboot my machine and try again.  Then i'll go back to testing a few other things with the slow builds times.
Comment 14 Joey Z 2017-02-02 23:37:06 UTC
Also, I'm running everything from my main solid state drive as well.
Comment 15 dean.ellis 2017-02-03 12:46:26 UTC
what we need is the results which include a performance summary

msbuild <proj> /t:Build /v:d /clp:PerformanceSummary



depending on the version msbuild. 

This will give you ALL the timings for each Task and will allow us to specifically find the problem task.
Comment 16 Joey Z 2017-02-03 18:37:28 UTC
Here it is.  Let me know if you need anything else


It contains the script I ran, the console output, and the generated text files, here's a sample command:
time msbuild Stringify.Core/Stringify.Core.csproj /t:Build /v:d /clp:PerformanceSummary > "$HOME/Desktop/JoeyZIncrementalBuild_Stringify.Core_Stringify.Core.txt"
Comment 17 dean.ellis 2017-02-07 10:23:16 UTC
Looking at the output GetAdditionalResourcesFromAssemblies is the problem area which is exactly what I expected. 

This is related to the fix [1]. This fix is now in the alpha channel (possibly the beta by now) and will be included in the next release. 

What is happening is the .sha1 hash is being calculated every build to check for zip integrity. The fix is to cache the hash. Note this will only effect the latest Support library nuget packages as they were the first release to include a SHA which the build system could check against. 

As mentioned the latest Alpha has the fix. Once the sha1 has been calculated and cached it will be used after that. 

[1] https://github.com/xamarin/xamarin-android/commit/2f690ca7d0c18055b01649de19b596df1626d3e2

[2] https://bugzilla.xamarin.com/show_bug.cgi?id=47288

*** This bug has been marked as a duplicate of bug 47288 ***
Comment 18 Joey Z 2017-02-07 18:29:06 UTC
Thanks for digging in and fixing it!  I'm excited to try it out.