Bug 43565 - [Cycle 8 Beta] Visual Studio often hangs (pauses, freezes) during `TastyConfig.DebugLaunch()` on second deployment attempt
Summary: [Cycle 8 Beta] Visual Studio often hangs (pauses, freezes) during `TastyConfi...
Status: VERIFIED FIXED
Alias: None
Product: Visual Studio Extensions
Classification: Xamarin
Component: iOS (show other bugs)
Version: 4.2.0 (C8)
Hardware: PC Windows
: High critical
Target Milestone: 4.2.0 (C8)
Assignee: Adrian Alonso
URL:
Depends on:
Blocks: 40959
  Show dependency tree
 
Reported: 2016-08-21 00:30 UTC by Brendan Zagaeski (Xamarin Support)
Modified: 2016-09-16 09:17 UTC (History)
6 users (show)

See Also:
Tags: BZRC8B1_C7SR1S1
Is this bug a regression?: Yes
Last known good build: Cycle 7 – Service Release 1


Attachments
Minidump of all VS threads during the hang (without heap) (2.89 MB, application/octet-stream)
2016-08-21 00:30 UTC, Brendan Zagaeski (Xamarin Support)
Details
Minimized from-scratch test case (17.58 KB, application/zip)
2016-08-25 02:37 UTC, Brendan Zagaeski (Xamarin Support)
Details

Description Brendan Zagaeski (Xamarin Support) 2016-08-21 00:30:31 UTC
Created attachment 17142 [details]
Minidump of all VS threads during the hang (without heap)

[Cycle 8 Beta] Visual Studio often hangs (pauses, freezes) during `TastyConfig.DebugLaunch()` on second deployment attempt


The Main thread call stack I get for these hangs looks roughly similar to Bug 39034, but that could be a false clue since that bug was reported against older versions where I am _not_ able to replicate this new bug.




## Steps to replicate

Use the same steps to replicate as described in non-public comment Bug 40959, Comment 3.

In summary, deploy the private test case from that bug once (with or without debugging) to the iOS simulator, stop debugging, and then attempt to redeploy it again by tapping Control-F5.

To reiterate some of the observations from that bug, the environment I'm using where I'm running Visual Studio in Windows 8.1 via a VMWare Fusion on the Mac build host itself might be important.


### Note

I recommend testing against Xcode 7.3, with iOS 9.3 simulators.  I ran into a quite consistent problem where the iOS 10 simulator itself froze if I already had it running when the Designer agent started, but that problem is demonstrably a _separate_ bug because the behavior of _this_ bug also happens on the Xcode 7.3 iOS 9.3 simulators, and those don't freeze.




## Regression status: regression in Cycle 8 Beta


### BAD

XamarinVS 4.2.0.508 (b9b610a)
Xamarin.iOS 9.99.3.8 (cycle8: 9eca876)


### BAD 

4.1.2.9-cycle7-buildsignature+a088786
(This was a testing development build from non-public comment Bug 40959, Comment 10)


### GOOD

XamarinVS 4.1.2.7  (af04e96060d9405335801d466beffd78c0b02de5) (Cycle 7 – Service Release 1 Beta)
XamarinVS 4.1.2.18 (fcbe082830a3736afb7d85537e5302b859bfa4c3) (Cycle 7 – Service Release 1 Stable)




## Results



### Repeatability


- I hit the problem on 7/7 trials on XamarinVS 4.2.0.508 (b9b610a).  At least in my particular environment, it never took more than 4 consecutive deployment attempts to hit the hang.

(In 2 of the trials, I used an existing build of the project from Cycle 7 SR 1.  In those trials, I deployed to Xcode 7.3, iOS 9.3 simulators.  For the other trials, I deployed to Xcode 8 Beta 5, iOS 10 simulators.)


- I am fairly certain that I _never_ hit the problem on any of the well over 20 trials I completed when testing Bug 40959 against older versions of XamarinVS.



### Call stack of Visual Studio's Main thread

> Microsoft.Build.dll!Microsoft.Build.Shared.ErrorUtilities.ThrowInvalidOperation(string resourceName, object[] args)	Unknown
> Microsoft.Build.dll!Microsoft.Build.Execution.BuildManager.BeginBuild(Microsoft.Build.Execution.BuildParameters parameters)	Unknown
> Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.PrepareDesignTimeBuild()	Unknown
> Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.AcquireBuildResources(Microsoft.VisualStudio.Shell.Interop.VSBUILDMANAGERRESOURCE fResources, out uint phCookie)	Unknown
> Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.CommonIDE.BuildManager.BuildManagerAccessor.BeginDesignTimeBuild()	Unknown
> Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.Build.ComInteropWrapper.ProjectShim.BuildTargetsImpl(string[] targetNamesToBuild, Microsoft.VisualStudio.Build.ComInteropWrapper.ITargetOutputs targetOutputs, bool retainCachedProjectInstance, bool isRealBuild, bool isOutOfProcBuild, Microsoft.VisualStudio.Build.ComInteropWrapper.IProjectShimBuildTargetsAsyncCallback callback)	Unknown
> Microsoft.VisualStudio.CommonIDE.dll!Microsoft.VisualStudio.Build.ComInteropWrapper.ProjectShim.BuildTargetListWithOutputs(string targetList, Microsoft.VisualStudio.Build.ComInteropWrapper.ITargetOutputs targetOutputs, bool isRealBuild, bool isOutOfProcBuild)	Unknown
> [Native to Managed Transition]	
> WindowsBase.dll!System.Windows.Threading.DispatcherSynchronizationContext.Wait(System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout)	Unknown
> mscorlib.dll!System.Threading.SynchronizationContext.InvokeWaitMethodHelper(System.Threading.SynchronizationContext syncContext, System.IntPtr[] waitHandles, bool waitAll, int millisecondsTimeout)	Unknown
> [Native to Managed Transition]	
> mscorlib.dll!System.Threading.WaitHandle.InternalWaitOne(System.Runtime.InteropServices.SafeHandle waitableSafeHandle, long millisecondsTimeout, bool hasThreadAffinity, bool exitContext)	Unknown
> mscorlib.dll!System.Threading.WaitHandle.WaitOne(int millisecondsTimeout, bool exitContext)	Unknown
> mscorlib.dll!System.Threading.WaitHandle.WaitOne()	Unknown
> Microsoft.Build.dll!Microsoft.Build.Execution.BuildSubmission.Execute()	Unknown
> Xamarin.VisualStudio.dll!Xamarin.VisualStudio.MsBuild.MsBuildService.BuildSubmisson(Microsoft.VisualStudio.Shell.Interop.IVsBuildManagerAccessor accessor, Microsoft.Build.Execution.BuildSubmission submission, bool buildInProgress, Microsoft.Build.Framework.ILogger[] loggers)	Unknown
> Xamarin.VisualStudio.dll!Xamarin.VisualStudio.MsBuild.MsBuildService.RunTarget(Clide.Solution.IProjectNode projectNode, string msbuildTarget, Xamarin.VisualStudio.IProgressReport progress, System.Collections.Generic.IDictionary<string,string> globalProperties, System.Collections.Generic.IDictionary<string,string> properties)	Unknown
> Xamarin.VisualStudio.dll!Xamarin.VisualStudio.MsBuild.MsBuildService.RunTarget(Clide.Solution.IProjectNode projectNode, string msbuildTarget, System.Collections.Generic.IDictionary<string,string> globalProperties, System.Collections.Generic.IDictionary<string,string> properties)	Unknown
> Xamarin.VisualStudio.IOS.dll!Xamarin.VisualStudio.IOS.MonoTouchProjectProperties.GetAppBundleDir(Xamarin.VisualStudio.IProgressReport progress)	Unknown
> Xamarin.VisualStudio.IOS.dll!Xamarin.VisualStudio.IOS.MonoTouchFlavoredProject.GetRunSessionInfo(Xamarin.VisualStudio.IOS.MonoTouchDevice device)	Unknown
> Xamarin.VisualStudio.IOS.dll!Xamarin.VisualStudio.IOS.MonoTouchFlavoredProject.RunApplication(Xamarin.VisualStudio.IOS.MonoTouchDevice device, Xamarin.VisualStudio.IProgressReport progress)	Unknown
> Xamarin.VisualStudio.IOS.dll!Xamarin.VisualStudio.IOS.MonoTouchFlavoredProject.DebugLaunch(Microsoft.VisualStudio.Shell.Interop.__VSDBGLAUNCHFLAGS grfLaunch, ref Xamarin.VisualStudio.IProgressReport progress)	Unknown
> Xamarin.VisualStudio.IOS.dll!Xamarin.VisualStudio.IOS.MonoTouchFlavoredProject.DebugLaunch(Microsoft.VisualStudio.Shell.Interop.__VSDBGLAUNCHFLAGS grfLaunch)	Unknown
> Xamarin.VisualStudio.dll!Xamarin.VisualStudio.TastyConfig.DebugLaunch(uint grfLaunch)	Unknown




## Experiment: Does the hang ever recover?

I left Visual Studio in the non-responsive state for 15 minutes on one of the trials.

### Results

No, the hang does not never recover.  Visual Studio did _not_ become responsive again at any point during that time, so I had to force-kill the process.




## Experiment: Revert the SSH.NET version

One of the changes introduced in a088786 (where I first observed this hang behavior) was to change the version of `Renci.SshNet` from:

2014.4.6-beta2

to:

2016.0.0-beta2

As an experiment, I applied the reverse of the trick mentioned in Bug 42411, Comment 0 to downgrade the SSH.NET assembly back to the older version.


### Results

Changing the SSH.NET version had no effect.  I continued to hit the same hangs.
Comment 2 Brendan Zagaeski (Xamarin Support) 2016-08-21 03:46:57 UTC
## The `.suo` file is apparently important

In further testing, I tried resetting my environment to start "from scratch" with the private .zip test case from Bug 40959.  On the first run after I had re-extracted the .zip file, I did not hit the problem.  But after I closed and reopened the solution, I started hitting the problem again.

As a follow-up test, I then quit Visual Studio, deleted the `.suo` file, and reopened the project.  The problem stopped again.

This dependency on the `.suo` file sounds very similar to Bug 39034.  See in particular Bug 39034, Comment 0:

> it works again when I delete the .vs folder



So it seems possible that the commits that were added between the GOOD and BAD versions (as recorded in Comment 0) were actually _not_ the direct cause of the problem.  Maybe instead those commits are themselves "good" but they are just _triggering_ an underlying problem that was already present in some way in the older versions (Bug 39034).




## Experiment: Are the contents of the particular solution important?

Yes.  I was unable to replicate the problem using a new template "Blank App" or "Single View App".

Note that the application project _itself_ is not critical to this problem: the application project in the private test case for Bug 40959 has no dependencies on any of the other projects in the solution (it is essentially an empty project), and yet if I remove all of the other "unused" items from the solution, I can no longer replicate the problem.




## Experiment: Is it important to use a resource-constrained environment (Windows running in a VM on the Mac build host itself) to hit the problem?

No.  I was able to hit this hang just as easily on a physical Windows machine as on a Windows VM.  (That is a noticeably different result compared to Bug 40959.)
Comment 3 Brendan Zagaeski (Xamarin Support) 2016-08-21 04:07:14 UTC
## Experiment: What happens if I delete the Xamarin.Forms.IntelliSense extension?

**It stops the problem**



I noticed that some of the "Worker Threads" in the minidump attached to Comment 0 show various method calls originating in "Xamarin.VisualStudio.Forms.Intellisense.dll", so I tried removing the Xamarin.Forms.IntelliSense extension.



1. Navigate to the Xamarin extensions folder (example for Visual Studio 2013):

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\Extensions\Xamarin\Xamarin\4.2.0.508



2. Move the Forms.IntelliSense files to a backup location:

Xamarin.VisualStudio.Forms.Intellisense.dll
Xamarin.VisualStudio.Forms.Intellisense.pdb
Xamarin.VisualStudio.Forms.Intellisense.pkgdef



3. In an administrator `cmd.exe` command prompt, run the following command to refresh Visual Studio's list of installed extensions:

"C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\devenv.exe" /setup /nosetupvstemplates



### Results

After performing these steps, I was _no longer able to hit the hang_, even after many repetitions of closing and reopening the solution and deploying the app.
Comment 4 Brendan Zagaeski (Xamarin Support) 2016-08-25 02:35:45 UTC
## Verification status: not yet resolved by candidate fix from Bug 39034, Comment 9.

### BAD: Cycle 8 Beta 2
XamarinVS 4.2.0.584 (918421b54a0f4f8a01df9f2a71298ce71f2f2708)
Xamarin.iOS 9.99.4.23 (f63ecd7)


Unfortunately, it appears that the candidate fix from the other similar-looking bug has not resolved the behavior of this bug.




## Further minimized "from scratch" test case

Since this bug is relatively easy to replicate, I have now taken the opportunity to further minimize the test case.

I was able to replicate the problem in a brand new solution using template projects (attached).



### Key results from the minimization



- It appears to be essential to have an unresolved library reference in the Xamarin.Forms PCL projects.  In the attached test case, I used the name "Foo".

To make a wild guess, the core of the problem might be in the "Worker Thread" call stack that has the following method at the top:
> Clide.dll!Clide.Solution.IProjectNodeExtensions.TryLoad(Microsoft.VisualStudio.Shell.Design.VsTargetFrameworkProvider provider, VSLangProj.Reference reference)


- Having 2 PCLs in the solution was not strictly required to hit the problem, but it seemed to make it noticeably easier, so I kept them both.



## Steps to replicate with the new test case

1. Open the attached test case in Visual Studio (tested in Visual Studio 2013 Update 5).

2. Wait for Visual Studio to fully connect to the Mac build host.

3. Set the "UnifiedSingleViewIphone1" project as the startup project, and set the configuration to "Debug|iPhoneSimulator".

4. Close and save the solution.

5. Reopen the solution via "File > Recent Projects and Solutions".

6. Choose "Debug > Start Without Debugging".

7. In my tests the build would often fail at this step because of the "unrelated" "BuildConnection" issue mentioned below (apparently due to leftover `MSBuild.exe` processes), so I quit and reopened Visual Studio at this step to get around the error.

8. Once the project is open and Visual Studio is re-connected to the build host, set the configuration to "Debug|iPhoneSimulator" again, and repeat step 6.

9. Wait for Visual Studio to finish building, deploying, and launching the app.

10. If Visual Studio isn't stuck yet, repeat step 6 several more times (up to 12 times).




## Repeatability

- In 6/6 "unclean" trials (where I had some leftover `MSBuild.exe` processes running) I hit a hang during step 8.



### Extra trials always starting from a completely "clean" state

Since the behavior of this bug appears to differ depending on whether there are any leftover `MSBuild.exe` processes running, I tried 3 additional trials where I always manually ended the leftover `MSBuild.exe` processes via Task Manager before running through the steps to replicate.

- In 3/3 trials I was able to hit the hang at step 10.  In each case I saw the problem after less than 12 repetitions of step 6.  In one case I even hit it on the first launch attempt.




## Unrelated issues


### 'BuildConnection'

In 6/6 "unclean" trials (and 0/3 "clean" trials) I hit an error about the 'BuildConnection' object at step 6:

> Cannot access a disposed object.
> Object name: 'BuildConnection'.

To work around this problem I quit and restarted Visual Studio.  I only saw this issue when there were leftover `MSBuild.exe` processes.  And unlike the Bug 42717, I'm not sure there's any way to avoid the leftover `MSBuild.exe` processes in this case because I was always force-quitting Visual Studio (so VS was never getting get a chance to do any cleanup).


### "ValidateAppBundleTask has been cancelled"

In 1/6 "unclean" trials I hit another different error after quitting and reopening Visual Studio:

> 'The post for client build6024Windo on topic
> xvs/Build/4.2.0.584/execute-task/UnifiedSingleViewIphone1/dc4e295%2FValidateAppBundleTask
> has been cancelled"

To solve this problem I quit Visual Studio, manually ended all of the running `MSBuild.exe` processes in Task Manager, and then restarted Visual Studio.
Comment 5 Brendan Zagaeski (Xamarin Support) 2016-08-25 02:37:00 UTC
Created attachment 17208 [details]
Minimized from-scratch test case

Note You need to log in before you can comment on or make changes to this bug.