Bug 60583 - Monitor.TryEnter and Process.WaitForExit incorrectly time out if system clock is adjusted forward
Summary: Monitor.TryEnter and Process.WaitForExit incorrectly time out if system clock...
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: io-layer ()
Version: 5.4 (2017-06)
Hardware: PC Windows
: --- normal
Target Milestone: ---
Assignee: Bugzilla
Depends on:
Reported: 2017-11-08 22:46 UTC by Brandon White
Modified: 2017-11-09 15:57 UTC (History)
2 users (show)

Is this bug a regression?: ---
Last known good build:

Minimally reproducing program for Monitor.TryEnter (965 bytes, text/csscript)
2017-11-08 22:46 UTC, Brandon White
Minimally reproducing program for Process.WaitForExit (633 bytes, text/csscript)
2017-11-08 22:47 UTC, Brandon White

Notice (2018-05-24): bugzilla.xamarin.com is now in read-only mode.

Please join us on Visual Studio Developer Community and in the Xamarin and Mono organizations on GitHub to continue tracking issues. Bugzilla will remain available for reference in read-only mode. We will continue to work on open Bugzilla bugs, copy them to the new locations as needed for follow-up, and add the new items under Related Links.

Our sincere thanks to everyone who has contributed on this bug tracker over the years. Thanks also for your understanding as we make these adjustments and improvements for the future.

Please create a new report for Bug 60583 on GitHub or Developer Community if you have new information to add and do not yet see a matching new report.

If the latest results still closely match this report, you can use the original description:

  • Export the original title and description: GitHub Markdown or Developer Community HTML
  • Copy the title and description into the new report. Adjust them to be up-to-date if needed.
  • Add your new information.

In special cases on GitHub you might also want the comments: GitHub Markdown with public comments

Related Links:

Description Brandon White 2017-11-08 22:46:26 UTC
Created attachment 25611 [details]
Minimally reproducing program for Monitor.TryEnter

Process.WaitForExit and Monitor.TryEnter incorrectly return with a failed timeout condition if the system clock is advanced forward by an amount equal to or more than the timeout duration.

This is because Mono is using sem_timedwait under the hood for these.  sem_timedwait unfortunately relies on CLOCK_REALTIME which is not a recommended way to express a timeout duration.  That's what CLOCK_MONOTONIC is for.

Recently, in Mono 5.2 major improvements were made to use CLOCK_MONOTONIC for the vast majority of timeout based APIs.  Anything using mono_os_cond_timedwait seems to now be immune to system clock changes (for platforms that support CLOCK_MONOTONIC).  This has been a very welcome and helpful improvement.

But there are still a few APIs that "break" if the system clock is advanced, which seems to be anything using mono_os_sem_timedwait.  Fortunately the surface area is much smaller now, but the issue remains nonetheless.

I'm hoping that Mono can completely avoid using sem_timedwait for platforms that support use of CLOCK_MONOTONIC using other means, like pthread_cond_timedwait.

There are also other internal uses of mono_os_sem_timedwait that I'm unable to determine if they exhibit erroneous or detrimental behavior when the system clock is advanced.

Attached are a couple of test programs that demonstrate the issue.  I've observed the problem on Debian 8 and 9 (on armhf and x64), and Ubuntu 16.04 (on x64).  Run the program in a console and observe a dot '.' output every second. Then from another console advance the system clock forward by more than 3 seconds.  The test program will output an error message in response to "timing out" on a blocking wait.

The end result is that if the system clock is moved forward, stuff errors out when it should not.
Comment 1 Brandon White 2017-11-08 22:47:03 UTC
Created attachment 25612 [details]
Minimally reproducing program for Process.WaitForExit
Comment 2 Brandon White 2017-11-08 23:13:30 UTC
CORRECTION -- Debian 8 does *not* exhibit the issue for me.  Has me scratching my head actually.
Comment 3 Brandon White 2017-11-09 15:57:39 UTC
I think I've gotten to the bottom of why Debian 8 does not have the issue, whereas Debian 9 and Ubuntu 16.04 do.

It boils down to the glibc version.  I think this commit [https://github.com/bminor/glibc/commit/c2f5813ae0a68f6c6d69e66dac2da6e46b9df034] is the critical change.  That shipped with glibc 2.22.

Using `ldd --version` you can check your glibc version.

Debian 8: 2.19
Debian 9: 2.24
Ubuntu 16.04: 2.23

The nature of the glibc change was to force use of CLOCK_REALTIME to conform with POSIX requirements.  As best I can tell, this resulted in sem_timedwait no longer behaving monotonically.  From the point of view of glibc, I think, that was a bug and this change fixed it.  So presumably Mono is relying on buggy behavior that got fixed.