Bug 21705 - "Compiling to native code..." step is out of the nominal build order/scope w.r.t. add-in notification.
Summary: "Compiling to native code..." step is out of the nominal build order/scope w....
Alias: None
Product: Xamarin Studio
Classification: Desktop
Component: General ()
Version: unspecified
Hardware: Macintosh Mac OS
: Normal normal
Target Milestone: master
Assignee: Bugzilla
Depends on:
Reported: 2014-07-30 12:44 UTC by Brian Berry
Modified: 2017-01-14 20:20 UTC (History)
4 users (show)

Is this bug a regression?: ---
Last known good build:

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report on Developer Community or GitHub with your current version information, steps to reproduce, and relevant error messages or log files if you are hitting an issue that looks similar to this resolved bug and you do not yet see a matching new report.

Related Links:

Description Brian Berry 2014-07-30 12:44:43 UTC
Found: Xamarin.iOS performs native code generation (the "mtouch" process) after the nominal build completion, resulting in out-of-order operations issues with custom XS add-ins.

Steps to reproduce:
   (1) Build any sample Xamarin.iOS app.
   (2) Observe the build output, e.g.:
          . . .
          Performing main compilation...
          . . .
          Build complete -- 0 errors, 1 warning
          . . .
          Compiling to native code
          /Developer/MonoTouch/usr/bin/mtouch -sdkroot....
          . . .

Expected: When a build is complete, all steps should be complete.

Additional details:  This is normally a non-issue, but we have a custom XS add-in in the mix here to accomplish some additional tasks specific to our builds/assets/etc.  The impact of the out-of-bounds build step is as follows.

In our XS plugin, we override ProjectServiceExtension.Build to set some things up prior to the nominal build (and clean up afterward), e.g. (simplified):

protected override BuildResult Build(IProgressMonitor monitor, SolutionEntityItem item, ConfigurationSelector cs)
   var br = new BuildResult();

   // Do some setup.

   br = base.Build(monitor, item, cs);

   // Do some cleanup.

   retur br;

The "Build complete -- 0 errors, 1 warning" notification comes during the call to base.Build(), as expected---by the time base.Build() returns we should expect the build to be complete.   Unfortunately, the "Compiling to native code" step does not occur until after the full Build() override returns...perhaps it has been dispatched to the GUI thread queue or otherwise deferred?

The net result is that our add-in related cleanups occur prior to this final build step and result in build failures.  To avoid this, we are having to do some very unfortunate hackery with extended properties to enact a dirty "clean up way later" mechanism.

Requesting that the mtouch step be scoped back with the true build, so that we know for sure that everything is done by the time the base Build() call returns!

(NOTE: If you know of another "we're really really done now...we swear!" event/method we can hook to get this notification, please feel free to comment here and we'll try that instead of the current remedies.)

Many thanks, everyone!
Comment 1 Jeffrey Stedfast 2014-07-30 14:54:34 UTC
I just double-checked and the mtouch "Compiling to native code..." phase is done in the Build() step.

The problem, I think, is that you are not inheriting from IPhoneBuildExtension - you are inheriting from ProjectServiceExtension, so calling base.Build() is probably not doing what you think it's doing.

AFAIK, the ProjectServiceExtensions get chained, so calling base won't necessarily call the iOS Build() logic.

Based on your output, I suspect that the iOS Build() logic, when it calls base.Build(), is chaining to *your* Build() method rather than the other way around.
Comment 2 Brian Berry 2014-07-30 17:33:40 UTC
That's certainly interesting---and I will check that.  This add-in is, however, not iPhone specific.  I'll certainly look into some kind of clean refactoring given your comments.   Thanks.
Comment 3 Jeffrey Stedfast 2014-07-30 18:06:21 UTC
I'm told that you don't actually need to subclass IPhoneBuildExtension, you just need to list the MonoDevelop.IPhone addin as a dependency.
Comment 4 Brian Berry 2014-07-30 19:40:58 UTC
A quick subclassing (plus revocation of our prior appeasements) did show the proper behavior, built the other targets without apparent issue.

Thanks for getting more words re: the dependency path---that sounds a lot better, and I will try it.
Comment 5 Brian Berry 2014-07-30 19:55:28 UTC
Quick update:  the dependency path did not result in desired behavior.  (If you meant simply adding it to the addin.xml e.g.:

                . . .
		<Addin id="::MonoDevelop.IPhone" version="5.2"/>

I will pursue it further, in case I missed something.
Comment 6 Brian Berry 2014-07-30 20:29:12 UTC
And I should note that, while improved from the standpoint of our immediate blocker, the subclassing approach results in duplication of work, e.g.:

Build complete -- 0 errors, 1 warning

Compiling to native code
/Developer/MonoTouch/usr/bin/mtouch -sdkroot . . .
BootstrapiOS.app built successfully.

Updating application manifest

Updating debug configuration file

Updating debug settings manifest

Embedding provisioning profile

Processing entitlements file

Preparing resources rules

Signing application
codesign -v -f -s . . .

*** BASE BUILD END ***  <--- my addition after base.Build(...) call in our addin.

Updating debug configuration file

Updating debug settings manifest

Embedding provisioning profile

Processing entitlements file

Preparing resources rules

Signing application
codesign -v -f -s ...

...which results in an additional note from the build that it's replacing an existing signature.

So, apparently a harmless outcome in this case.  Architecturally inelegant, however.  Prone to breakage in the future as well.   Will hunt further.
Comment 7 Mikayla Hutchinson [MSFT] 2014-07-30 21:05:15 UTC
FWIW our MSBuild targets for iOS will be released soon, and will make the build process much more extensible and customizable.
Comment 8 Brian Berry 2014-07-31 10:12:03 UTC
That will certainly be interesting to look for, Michael...thanks.

I was thinking about this further overnight, and wonder if the addition of two simple events would suffice to provide add-in authors a means to do what they want before/after a build without having to mess with the build override itself.   We have, for example, done the same extension work in VS as well as XS, via a VSPackage.   To accomplish the same thing there, you hook begin/end events at two scope levels that encompass VS's activities, e.g.:

OnBuildBegin(vsBuildScope scope, vsBuildAction action)
    . . .
            . . . (VS does the project builds/cleans/rebuilds here) . . .
        . . .
            . . .
    . . .
OnBuildDone(vsBuildScope scope, vsBuildAction action)

Of course, I'm sure you are well aware of that pattern there---as you have your own customizations to VS.

What would likely solve our issues (by not having to override/mess with the Build() at all) is having similar events to at least the inner scope (OnBuildProjConfigBegin/End equivalents) by which we can perform true pre-build and true post-build activities without having to add per-project tasks to the .csproj files everywhere to manually invoke them.  (Currently our tooling is essentially free from developer intervention.)   The outer scope would be useful in cases I can envision (e.g., if you needed to tweak project-project dependencies for build order, etc.).  But, to us, the inner ones are where the rubber currently meets the road.

Anyhow, food for thought.   We appear to be unblocked by hijacking IPhoneBuildExtension vs ProjectServiceExtension (at least for the moment).  If there are equivalents for OSX+Android that we need to honor, we'll factor everything into separate service extensions and hope for the best.

As always---thanks for taking the time to assist!
Comment 9 Jeffrey Stedfast 2014-08-14 14:38:39 UTC
Moving this to General because it's not really iOS-specific
Comment 10 Brian Berry 2014-10-19 14:44:27 UTC
I'd like to ask for further guidance on this issue, the proper way to achieve via an addin the ability to, for any given platform via an addin, given the existing BuildResult Build(IProgressMonitor monitor, SolutionEntityItem item, ConfigurationSelector cs) method:
  (1) Perform work just prior to the nominal platform-specific stuff done in the inherited Build().
  (2) Allow the default Build() operation to then occur.
  (3) Be able to perform post-build operations/cleanup once Build() completes.

I am now in a position where my initial fixes (including a subclassing of IPhoneBuildExtension) no longer result in proper (and worse: nondeterministic) behavior.

Again, without pre-build/post-build outer events like MSVS, I am having to do my prebuild/postbuild work within a Build() override, calling base.Build() in between.   This does not result in a proper order of operations, including bizarre issues like:
  * Duplicate build phases (signing identity checks, compile-to-native-code steps, ...)
  * Some of these duplicates occur *after* my cleanup step in the override Build(), resulting in errors.  It puzzles me, as if the base extension is not doing all of its build work in the context of its own Build method()---as if a series of build operations are somehow queued instead, of which Build() is only one instead of it being atomic.

If an extension cannot override Build() (chaining to base) given the underlying architecture, how can we create an extension that automatically performs pre/post build operations of our own design *without* having to edit each and every project to add custom build steps?  These custom operations are intended to be implicit for all projects using the extension, no matter the project.

(If I am not being clear, please let me know and I will take another swing. :)
Comment 11 Brian Berry 2014-10-19 17:44:57 UTC
Additional comments, after extensive logging/empirics.  :)

I think part of the issue here is that interaction of multiple service extensions (SE), including those packaged with XS for its purposes is a bit unclear.

A first exploration of SE authoring will typically have you at the search endpoint http://monodevelop.com/Developers/Articles/How_to_extend_the_Project_Model#Project_service_extensions, which adequately describes the simple, single-SE case (do some work before/after the build, call baseBuild() to have the normal build take place).

From my logging (and earlier comments made by others, too), it's apparent that the SEs registered with my addin are logically subclassed in registration order (???).  That is, for a bland first SE,an SE for Android, an SE for iOS, calls to base.Build(...) at each level end up calling the Build() for the next SE in that chain.   This is in apparent contrast to what I would expect---calling the Build() for the *true* base class for said extension.

I'll have to revisit the MonoDevelop core code some more, but documentation on this chaining---if not already present---would be pretty valuable.  It's not obvious---and with knowledge that each will be called in turn, folks will need to know to dodge execution by project type if an SE is meant only for iOS projects, for instance.

(Another aside: rules/consequences for subclassing existing packaged SEs like IPhoneBuildExtension/XamMac2BuildExtension/etc.?)

I'll bumble my way to correct operation (again) in the meantime.  :)
Thanks, all.
Comment 14 Brian Berry 2017-01-14 20:20:00 UTC
FYI, this is the no-longer-care bucket here as we have converted all related customizations from extension-space to msbuild task-space.  Marking resolved/answered.