Bug 43188 - Sporadic build error (Out of memory) leaves solution in unbuildable state until reload of project.
Summary: Sporadic build error (Out of memory) leaves solution in unbuildable state unt...
Status: VERIFIED FIXED
Alias: None
Product: Android
Classification: Xamarin
Component: General (show other bugs)
Version: 7.0 (C8)
Hardware: Macintosh Mac OS
: Highest major
Target Milestone: 7.1 (C9)
Assignee: Jonathan Pryor
URL:
: 43505 44556 45005 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-08-09 11:38 UTC by Michael Sivers
Modified: 2017-05-02 19:54 UTC (History)
19 users (show)

Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Ide log file (82.08 KB, text/plain)
2016-08-09 11:38 UTC, Michael Sivers
Details


Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.


Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:
Status:
VERIFIED FIXED

Description Michael Sivers 2016-08-09 11:38:25 UTC
Created attachment 16953 [details]
Ide log file

Description of Problem:

Xamarin Studio 6.0.1 (build 9)

Sporadic Build Error. Everything is building fine then sporadically will get the following series of errors. Once this happens (a couple of times a day so far) the only way to resolve it is to close the project and re-open it, then builds work fine.

Please note that I have attached the 'Ide' log file.


***************************
INITIALLY GET THIS ERROR...
***************************

Error: /Users/msivers/SystemsTFS/MobileApps/Main/Apps/Next.Droid/Next.Droid.csproj: ../../Shared/Next.Shared/Next.Shared.projitems: Project file could not be imported, it was being imported by /Users/msivers/SystemsTFS/MobileApps/Main/Apps/Next.Droid/Next.Droid.csproj: Out of memory (Next.Droid)

Building: Next.Droid (Debug)

Build started 09/08/2016 11:45:55.
__________________________________________________
Project "/Users/msivers/SystemsTFS/MobileApps/Main/Shared/Next.Localization/Next.Localization.csproj" (Build target(s)):
	
	Target PrepareForBuild:
		Configuration: Debug Platform: AnyCPU
	
	Target GenerateSatelliteAssemblies:
	No input files were specified for target GenerateSatelliteAssemblies, skipping.
	
	Target GenerateTargetFrameworkMonikerAttribute:
	Skipping target "GenerateTargetFrameworkMonikerAttribute" because its outputs are up-to-date.
	
	Target CoreCompile:
	Skipping target "CoreCompile" because its outputs are up-to-date.
Done building project "/Users/msivers/SystemsTFS/MobileApps/Main/Shared/Next.Localization/Next.Localization.csproj".

Build succeeded.
	 0 Warning(s)
	 0 Error(s)

Time Elapsed 00:00:00.0514500

---------------------- Done ----------------------

Build: 1 error, 0 warnings




***********************************************************************
I THEN GET THE FOLLOWING ERROR FOR EVERY SUBSEQUENT ATTEMPT TO BUILD...
***********************************************************************

/Users/msivers/SystemsTFS/MobileApps/Main/Shared/Next.Localization/Next.Localization.csproj(1,1): Error: Unknown MSBuild failure. Please try building the project again (Next.Localization)


Building: Next.Droid (Debug)

---------------------- Done ----------------------

Build: 1 error, 0 warnings
Comment 1 Michael Sivers 2016-08-10 11:39:39 UTC
I have updated Xamarin Studio to latest 6.0.2 (build 73) and still have this happening sporadically.
Comment 2 Michael Sivers 2016-08-10 12:38:47 UTC
For reference, all our developers on Xamarin Studio (5) have experience this issue. We are experiencing this issue on average 1-2 times an hour and as the only workaround is to reload the solution, it is becoming a significant frustration.

Regards.
Comment 3 Mark Darling 2016-08-10 13:28:20 UTC
Same issue here -

Error executing task LinkAssemblies: Out of memory

.. on initial build, all subsequent builds from this point show...

Error Unknown MSBuild failure. Please try building the project again

I've found that mono-sgen32 was holding onto 3GB of memory, killing that process and the error went away. Restarting Xamarin Studio has the same effect and fixes the problem.

Happening multiple times per hour on 6.0.2 build 73
Comment 4 Aaron Bockover [MSFT] 2016-08-12 19:54:51 UTC
I am experiencing this as well on 6.1 (alpha channel). It's the 5th time today that I've had to restart XS.

  XS Version 6.1 (build 5308)
  Mono 4.6.0 (mono-4.6.0-branch/0cb79a3) (64-bit)

If I look at activity monitor, I can see:

  Main XS Process: 1.34GB
  mono-sgen32 child process: 3.11GB

The child process is presumably the MSBuild process? It has hundreds of open assembly file handles (we have dozens of projects in the solution, targeting *every* Xamarin framework (Android, iOS, Mac Mobile, Mac Desktop, etc) - this results in basically every single framework assembly from every single framework being opened by the process).

I assume that `mono-sgen32` is a 32 bit process, and so it really might be out of memory (approaching 4GB, the runtime/GC probably hits that limit even though the OS indicates slightly less physical memory used). For starters, this should probably be a 64 bit process?

However, something is definitely leaking. It takes maybe 20-30 builds for the limit to be hit.

It manifests as many "Unknown MSBuild failure. Please try building the project again" errors (with nothing of note in the actual build log). These errors are also accompanied with a generic "Out of memory" and "Error executing task MSBuild: Out of memory" errors in the error pane.

In the Ide.log we can see the following, presumably related:

ERROR [2016-08-12 15:32:05Z]: RunTarget failed
System.Exception: MSBuild operation failed ---> System.NullReferenceException: Object reference not set to an instance of an object
  at System.Xml.XmlEncodedRawTextWriter.FlushBuffer () [0x0009d] in <filename unknown>:0 
  at System.Xml.XmlEncodedRawTextWriter.Close () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlRawWriter.Close (System.Xml.WriteState currentState) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlWellFormedWriter.Close () [0x000b9] in <filename unknown>:0 
  at System.Xml.XmlWriter.Dispose (System.Boolean disposing) [0x00012] in <filename unknown>:0 
  at System.Xml.XmlWriter.Dispose () [0x00000] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.GenerateSolutionConfigurationContents (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations) [0x0011e] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.SetupProject (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations) [0x00002] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder+<Run>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.BuildEngine.STARunner () [0x00010] in <filename unknown>:0 
[MVID] 3a25cb2f0bd44e29b16f27d9ece78a5d 6,7,8,9
[MVID] 6f5b897a4e79420ea17c734c0583d6a2 0,1,2,3,4,5
   --- End of inner exception stack trace ---


Server stack trace: 
  at MonoDevelop.Projects.MSBuild.BuildEngine.RunSTA (System.Int32 taskId, System.Threading.ThreadStart ts) [0x00117] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.Run (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations, MonoDevelop.Projects.MSBuild.ILogWriter logWriter, MonoDevelop.Projects.MSBuild.MSBuildVerbosity verbosity, System.String[] runTargets, System.String[] evaluateItems, System.String[] evaluateProperties, System.Collections.Generic.Dictionary`2[TKey,TValue] globalProperties, System.Int32 taskId) [0x00049] in <filename unknown>:0 
  at (wrapper managed-to-native) System.Runtime.Remoting.RemotingServices:InternalExecute (System.Reflection.MethodBase,object,object[],object[]&)
  at System.Runtime.Remoting.RemotingServices.InternalExecuteMessage (System.MarshalByRefObject target, System.Runtime.Remoting.Messaging.IMethodCallMessage reqMsg) [0x000ca] in <filename unknown>:0 
[MVID] 3a25cb2f0bd44e29b16f27d9ece78a5d 0,1
[MVID] 7747cf446af449e194e4b4e70d85e773 3

Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) MonoDevelop.Projects.MSBuild.ProjectBuilder:Run (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[],MonoDevelop.Projects.MSBuild.ILogWriter,MonoDevelop.Projects.MSBuild.MSBuildVerbosity,string[],string[],string[],System.Collections.Generic.Dictionary`2<string, string>,int)
  at MonoDevelop.Projects.MSBuild.RemoteProjectBuilder+<Run>c__AnonStorey4.<>m__0 () [0x0000b] in /Users/builder/data/lanes/3509/fc88ad6d/source/monodevelop/main/src/core/MonoDevelop.Core/MonoDevelop.Projects.MSBuild/RemoteProjectBuilder.cs:241 
[MVID] 85b319cd37a34a86a692143b44331091 2
ERROR [2016-08-12 15:32:05Z]: RunTarget failed
System.Exception: MSBuild operation failed ---> System.NullReferenceException: Object reference not set to an instance of an object
  at System.Xml.XmlEncodedRawTextWriter.FlushBuffer () [0x0009d] in <filename unknown>:0 
  at System.Xml.XmlEncodedRawTextWriter.Close () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlRawWriter.Close (System.Xml.WriteState currentState) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlWellFormedWriter.Close () [0x000b9] in <filename unknown>:0 
  at System.Xml.XmlWriter.Dispose (System.Boolean disposing) [0x00012] in <filename unknown>:0 
  at System.Xml.XmlWriter.Dispose () [0x00000] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.GenerateSolutionConfigurationContents (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations) [0x0011e] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.SetupProject (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations) [0x00002] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder+<Run>c__AnonStorey0.<>m__0 () [0x00000] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.BuildEngine.STARunner () [0x00010] in <filename unknown>:0 
[MVID] 3a25cb2f0bd44e29b16f27d9ece78a5d 6,7,8,9
[MVID] 6f5b897a4e79420ea17c734c0583d6a2 0,1,2,3,4,5
   --- End of inner exception stack trace ---


Server stack trace: 
  at MonoDevelop.Projects.MSBuild.BuildEngine.RunSTA (System.Int32 taskId, System.Threading.ThreadStart ts) [0x00117] in <filename unknown>:0 
  at MonoDevelop.Projects.MSBuild.ProjectBuilder.Run (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[] configurations, MonoDevelop.Projects.MSBuild.ILogWriter logWriter, MonoDevelop.Projects.MSBuild.MSBuildVerbosity verbosity, System.String[] runTargets, System.String[] evaluateItems, System.String[] evaluateProperties, System.Collections.Generic.Dictionary`2[TKey,TValue] globalProperties, System.Int32 taskId) [0x00049] in <filename unknown>:0 
  at (wrapper managed-to-native) System.Runtime.Remoting.RemotingServices:InternalExecute (System.Reflection.MethodBase,object,object[],object[]&)
  at System.Runtime.Remoting.RemotingServices.InternalExecuteMessage (System.MarshalByRefObject target, System.Runtime.Remoting.Messaging.IMethodCallMessage reqMsg) [0x000ca] in <filename unknown>:0 
[MVID] 3a25cb2f0bd44e29b16f27d9ece78a5d 0,1
[MVID] 7747cf446af449e194e4b4e70d85e773 3

Exception rethrown at [0]: 
  at (wrapper managed-to-native) System.Object:__icall_wrapper_mono_remoting_wrapper (intptr,intptr)
  at (wrapper remoting-invoke) MonoDevelop.Projects.MSBuild.ProjectBuilder:Run (MonoDevelop.Projects.MSBuild.ProjectConfigurationInfo[],MonoDevelop.Projects.MSBuild.ILogWriter,MonoDevelop.Projects.MSBuild.MSBuildVerbosity,string[],string[],string[],System.Collections.Generic.Dictionary`2<string, string>,int)
  at MonoDevelop.Projects.MSBuild.RemoteProjectBuilder+<Run>c__AnonStorey4.<>m__0 () [0x0000b] in /Users/builder/data/lanes/3509/fc88ad6d/source/monodevelop/main/src/core/MonoDevelop.Core/MonoDevelop.Projects.MSBuild/RemoteProjectBuilder.cs:241 
[MVID] 85b319cd37a34a86a692143b44331091 2
Comment 6 Aaron Bockover [MSFT] 2016-08-12 20:14:40 UTC
"wall clock" sampling of mono-sgen32 child process after a build:

  Build #   Memory Used:
  1         255MB
  2         393MB
  3         662MB
  4         698MB
  5         987MB
  6         1.07GB
  7         1.30GB
  8         1.27GB
  9         1.58GB
  10        1.76GB
  11        1.87GB
  12        1.97GB
  13        2.02GB
  14        2.32GB
  15        2.41GB
  16        2.76GB
  17        2.80GB
  18        3.07GB
  19 *      3.10GB

The 19th build failed with OOM.
  
It's interesting that it seems like there's a big jump roughly every _other_ build.

XS itself also grows about 80MB/build.
Comment 7 David Dunscombe 2016-08-15 10:19:52 UTC
I also experience this, especially annoying as I also have to do a clean/rebuild after reload as fody onpropertychange generates an exception otherwise. Its done it for all 6.X builds of xamarin studio.
Comment 8 Greg Munn 2016-08-18 14:26:16 UTC
Possibly duplicated by bug #43505
Comment 9 dean.ellis 2016-10-04 15:27:19 UTC
*** Bug 45005 has been marked as a duplicate of this bug. ***
Comment 10 Lluis Sanchez 2016-10-06 21:17:37 UTC
It's a leak in Xamarin.Android. It doesn't happen on XA master. The fix is probably https://github.com/xamarin/xamarin-android/commit/16d65fab5e5c8606c727bea804bb40d105745d13 or https://github.com/xamarin/xamarin-android/commit/fa61551b9007be54c0163f52f5c7bb34d0d4d0ee, or both. It needs to be backported to the cycle8 branch.
Comment 12 xamarin-release-manager 2016-10-12 10:15:37 UTC
Fixed in version 6.2.0.812 (master)

Author: Lluis Sanchez
Commit: f6e5b6a4796687245fa628ba952be719677ed1f8 (mono/monodevelop)
Comment 13 xamarin-release-manager 2016-10-12 10:17:22 UTC
Fixed in version 6.1.2.29 (cycle8)

Author: Lluis Sanchez
Commit: 3dbc9bdffb352d0d14d8260a0d897c325c26e4e1 (mono/monodevelop)
Comment 14 Lluis Sanchez 2016-10-12 10:18:53 UTC
Reopening since the issue is not 100% fixed. The error will still happen, but it won't leave the project in an unbuildable state anymore.
Comment 15 Lluis Sanchez 2016-10-18 10:36:45 UTC
*** Bug 44556 has been marked as a duplicate of this bug. ***
Comment 16 Peter Collins 2016-10-28 15:52:13 UTC
We've taken the majority of these patches in C8SR1, which will provide a net improvement in experience. Moving this bug to the Android product and Cycle9 milestone to track the Cecil 0.10 update which is presumed to resolve this issue.
Comment 17 Jonathan Pryor 2016-11-14 20:20:24 UTC
Closing as FIXED because, to the best of my current knowledge, Cycle 9/Xamarin.Android 7.1 doesn't contain the specified memory leak.
Comment 18 Gonzalo Martin 2017-01-10 15:17:14 UTC
I could not reproduce the issue. Do you have any sample to attach where you are able to reproduce this bug? Thanks
Comment 19 Aaron Bockover [MSFT] 2017-02-17 15:26:06 UTC
I can no longer reproduce this issue in C9.
Comment 20 Peter Collins 2017-02-21 19:58:08 UTC
Marking as Verified/Fixed as per comment #19.
Comment 21 Jeffrey Stedfast 2017-05-02 19:54:43 UTC
*** Bug 43505 has been marked as a duplicate of this bug. ***