Bug 44629 - SocketAsyncResult IsComplete ObjectDisposedException
Summary: SocketAsyncResult IsComplete ObjectDisposedException
Status: NEEDINFO
Alias: None
Product: Class Libraries
Classification: Mono
Component: System (show other bugs)
Version: 4.4.2 (C7SR1)
Hardware: PC Mac OS
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
: 44628 (view as bug list)
Depends on:
Blocks:
 
Reported: 2016-09-22 16:33 UTC by Oliver
Modified: 2017-09-22 20:25 UTC (History)
7 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Oliver 2016-09-22 16:33:58 UTC
I've been having issues where I will encounter an exception which can't be caught which crashes my program.

Unhandled Exception:
 System.ObjectDisposedException: Cannot access a disposed object.
 Object name: 'System.Threading.ManualResetEvent'.
   at System.Threading.WaitHandle.CheckDisposed () [0x00043] in /mono/mcs/class/corlib/System.Threading/WaitHandle.cs:435 
   at System.Threading.EventWaitHandle.Set () [0x0000c] in /mono/mcs/class/corlib/System.Threading/EventWaitHandle.cs:205 
   at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Set ()
   at System.IOAsyncResult.set_IsCompleted (Boolean value) [0x00024] in /mono/mcs/class/System/System/IOSelector.cs:118 
   at System.Net.Sockets.SocketAsyncResult.Complete () [0x00044] in /mono/mcs/class/System/System.Net.Sockets/SocketAsyncResult.cs:146 
   at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in /mono/mcs/class/System/System.Net.Sockets/SocketAsyncResult.cs:214 
   at System.Net.Sockets.Socket.<BeginConnectCallback>m__4 (System.IOAsyncResult ares) [0x0012a] in /mono/mcs/class/System/System.Net.Sockets/Socket.cs:1579 
   at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /mono/mcs/class/System/System/IOSelector.cs:143 
   at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in /mono/external/referencesource/mscorlib/system/threading/threadpool.cs:857 
   at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /mono/external/referencesource/mscorlib/system/threading/threadpool.cs:1212

I managed to trace the issue down to a call that was made in the RabbitMQ.Client nuget package version 3.6.5 in the Connect method where AsyncWaitHandle.Close (which dipososes of the wait handle) is called. In some instances it appears this bit of code executes before the code called above and so an ObjectDisposedException occurs. I have tested this with mono 4.6 and have the same issue with a similar stacktrace, I'm not quite sure what the expected behaviour should be. I have been able to reproduce this issue by running rabbitmq, connecting to it and then pulling it down so the program can no longer connect, but it only happens every so often.

https://github.com/rabbitmq/rabbitmq-dotnet-client/blob/rabbitmq_v3_6_5/projects/client/RabbitMQ.Client/src/client/impl/SocketFrameHandler.cs

private void Connect(ITcpClient socket, AmqpTcpEndpoint endpoint, int timeout)
{
    IAsyncResult ar = null;
    try
    {
        ar = socket.BeginConnect(endpoint.HostName, endpoint.Port, null, null);
        if (!ar.AsyncWaitHandle.WaitOne(timeout, false))
        {
            socket.Close();
            throw new TimeoutException("Connection to " + endpoint + " timed out");
        }
        socket.EndConnect(ar);
    }
    catch (ArgumentException e)
    {
        throw new ConnectFailureException("Connection failed", e);
    }
    catch (SocketException e)
    {
        throw new ConnectFailureException("Connection failed", e);
    }
    finally
    {
        if (ar != null)
        {
            ar.AsyncWaitHandle.Close();
        }
    }
}
Comment 1 Marek Safar 2016-09-23 07:00:00 UTC
*** Bug 44628 has been marked as a duplicate of this bug. ***
Comment 2 Warren 2016-09-29 13:52:37 UTC
This also happens when calling BeginConnect() with a callback or when using ConnectAsync().  This appears to be a pretty serious bug.  It isn't clear to me how any asynchronous connections are possible in a production environment right now.

On Linux with mono 4.6.1, this behavior is intermittent, failing 90% of the time.

I believe it to be caused by a bug in ConnectAsync().  ConnectAsync() returns false (meaning that it completed synchronously and will not fire the Completed event, but then it calls the Completed event anyway resulting resulting in completion logic running twice).

The ConnectAsync() behavior is definitely a bug.  If it returns false, it must not fire the event, but it Mono it returns false whether it fires the event or not.

The following code demonstrates the problem:

using System;
using System.Net;
using System.Net.Sockets;
using System.Threading;
using System.Threading.Tasks;

namespace ConnectAsyncBug
{
    class Program
    {
        static void Main(string[] args)
        {
            var listener = new Socket(SocketType.Stream, ProtocolType.Tcp);
            var endpoint = new IPEndPoint(IPAddress.Loopback, 33444);
            listener.Bind(endpoint);

            listener.Listen(5);
            Task.Run(() =>
            {
                listener.Accept();
            });

            var socket = new Socket(SocketType.Stream, ProtocolType.Tcp);
            var connectArgs = new SocketAsyncEventArgs
            {
                RemoteEndPoint = endpoint,
            };
            int eventFireCount = 0;
            connectArgs.Completed += (sender, asyncArgs) =>
            {
                Interlocked.Increment(ref eventFireCount);
            };
            var willFireEvent = socket.ConnectAsync(connectArgs);
            Thread.Sleep(100);
            if (willFireEvent == false && eventFireCount > 0)
                Console.WriteLine("Wrong: will fire event returned {0} but the event fired {1} times anyway", willFireEvent, eventFireCount);
            else if (willFireEvent == true)
                Console.WriteLine("Could not test, conditions not met.  Will fire event returned {0}", willFireEvent);
            else if (willFireEvent == false && eventFireCount == 0)
                Console.WriteLine("Good: will fire event returned {0} and the event fired {1} times", willFireEvent, eventFireCount);
            else
                Console.WriteLine("{0}, {1}", willFireEvent, eventFireCount);
        }
    }
}

(Note: in Windows the event is always fired in both .Net and Mono.  .Net correctly returns true in this case, so we can't actually test the synchronous case.  Mono always returns false so we see the error every time.  In Linux, the event is fired about 90% of the time in my tests.  So when it returns false it is wrong 90% of the time, but is right 10% of the time.)
Comment 4 Matt Z 2016-11-30 03:40:39 UTC
We are seeing this frequently with Mono 4.6.2.7 on Linux/x86-64 as well, though @Warren's program is not reproducing the issue for us, even when called in a loop (it always hits the "Could not test, conditions not met" case).
Comment 5 Warren 2016-11-30 12:45:03 UTC
There is more than one way to cause this particular exception, which has led to some confusion.  There was a bug which my code was able to reproduce on linux but no longer does with the most recent versions - so I think this bug has been fixed.

However I have since seen the exception again when using ConnectAsync() and again only on Linux.  I was able to narrow the exception down to a place where I had done something like this:

    var connectTask = socket.ConnectAsync(...);
    using (...) {
        var connection = await connectTask;
    }

And then I don't get it if I do this:

    using (...) {
        var connection = await socket.ConnectAsync(...);
    }

But I don't know why or if this is similar to your problem.  I suspect a race condition somewhere which may be in my code and not Mono (although the stack trace never contains my code), but I am unsure at this time.  I don't know if the using() has anything to do with the issue, but I included it because that's the pattern I noticed - note nothing in the using clause is used in the call to ConnectAsync().  So I hope that helps.  Making the above change stopped the issue for me and I've never seen it again.
Comment 6 Matt Z 2016-12-02 00:05:32 UTC
Warren, I'm not sure I follow. If those two code snippets produced different effects, that would be a compiler bug, not a bug in your code.

The bug, from glancing at the related Mono code a bit now, seems to be caused from IOAsyncResult exposing it's private ManualResetEvent instance as a property, which lets outsiders then call Dispose() on it. IOAsyncResult is unaware that it's been disposed (and actually has no means of checking), and then later calls Set() which causes the ObjectDisposedException.

One potential fix would be to change all callers of IOAsyncResult.AsyncWaitHandle to never call Dispose() on the result. This seems correct to me since the handle is still owned by IOAsyncResult even though it exposes it externally.

Another potential fix would be to change IOAsyncResult.AsyncWaitHandle property to return a custom WaitHandle class that wraps the internal handle and disables Dispose()/Close().
Comment 7 Matt Z 2017-03-31 18:44:37 UTC
We were hoping since there was some refactoring in Mono 4.8.0.520 that this might have been fixed, unfortunately we still see it though the stack trace is a little different:

[ERROR] FATAL UNHANDLED EXCEPTION: System.ObjectDisposedException: Safe handle has been closed
  at System.Runtime.InteropServices.SafeHandle.DangerousAddRef (System.Boolean& success) [0x00025] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/corlib/System.Runtime.InteropServices/SafeHandle.cs:125
  at System.Threading.NativeEventCalls.SetEvent (Microsoft.Win32.SafeHandles.SafeWaitHandle handle) [0x00002] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/corlib/System.Threading/NativeEventCalls.cs:54
  at System.Threading.EventWaitHandle.Set () [0x00000] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/referencesource/mscorlib/system/threading/eventwaithandle.cs:318
  at (wrapper remoting-invoke-with-check) System.Threading.EventWaitHandle:Set ()
  at System.IOAsyncResult.set_IsCompleted (System.Boolean value) [0x00024] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/System/System/IOSelector.cs:118
  at System.Net.Sockets.SocketAsyncResult.Complete () [0x00044] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/System/System.Net.Sockets/SocketAsyncResult.cs:146
  at System.Net.Sockets.SocketAsyncResult.Complete (System.Exception e) [0x00007] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/System/System.Net.Sockets/SocketAsyncResult.cs:214
  at System.Net.Sockets.Socket.<BeginConnectCallback>m__4 (System.IOAsyncResult ares) [0x0012a] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/System/System.Net.Sockets/Socket.cs:1563
  at System.IOSelectorJob.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () [0x00000] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/System/System/IOSelector.cs:143
  at System.Threading.ThreadPoolWorkQueue.Dispatch () [0x00096] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:854
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () [0x00000] in /jenkins/workspace/Mono_Main/rpmbuild/BUILD/mono-4.8.0/mcs/class/referencesource/mscorlib/system/threading/threadpool.cs:1209

It seems avoiding Socket.ConnectAsync() helps avoid the bug in Mono somewhat, but we can't control package dependencies that call it (npgsql, rabbitmq.client, etc).
Comment 8 Marek Safar 2017-09-08 23:32:20 UTC
Thank you for your report!

I’m unable to reproduce this issue locally using the information you provided. In order to investigate the issue further can you please attach a reproduction project and steps to reproduce this issue? For help on writing a bug report, please see our guide on this topic:

https://bugzilla.xamarin.com/page.cgi?id=bug-writing.html

Some of your next steps would be:

1. Including a sample project or steps to reproduce this problem
2. Your Version Information
3. Your expected results and actual results

Thank you!

Note You need to log in before you can comment on or make changes to this bug.