Bug 55443 - SO_REUSEADDR does not produce expected results after Mono 4.2.1
Summary: SO_REUSEADDR does not produce expected results after Mono 4.2.1
Status: NEW
Alias: None
Product: Runtime
Classification: Mono
Component: io-layer (show other bugs)
Version: 4.8.0 (C9)
Hardware: All Linux
: --- normal
Target Milestone: ---
Assignee: Ludovic Henry
URL:
Depends on:
Blocks:
 
Reported: 2017-04-21 04:51 UTC by H3V2
Modified: 2017-04-24 13:46 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Program which demonstrates the reported issue (1.08 KB, text/plain)
2017-04-21 04:51 UTC, H3V2
Details

Description H3V2 2017-04-21 04:51:21 UTC
Created attachment 21672 [details]
Program which demonstrates the reported issue

Observed behaviour: When a socket is double bound to the same endpoint, an AddressAlreadyInUse exception is thrown
Expected behaviour: An exception should not be thrown

I originally posted this as a question on Stack Overflow, but further investigation leads me to believe that what I'm experiencing is a bug..

http://stackoverflow.com/questions/43526650/so-reuseaddr-does-not-produce-expected-results-after-mono-4-2-1


Reading back through the source code and issues I can see there has been a lot of confusion about SO_REUSEADDR, SO_REUSEPORT and ExclusiveAddressUse which I believe has led to this bug.

My issue is that using Mono on Linux after version 4.2.1, I am unable to Bind() a Socket to a LocalEndPoint if there is already a socket bound and listening to that same endpoint.

I've attached a sample program which demonstrates this phenomenon, and to save time i'll briefly re-cap the function of each of the moving parts involved.

  SO_REUSEADDR (0x0004) = allows binding to an address which is in a TIME_WAIT state.
  SO_REUSEPORT (0x0200) = allows multiple processes to bind to the same address provided all of them use the SO_REUSEPORT option.

Normally applications cannot bind to an address and port belonging to a recently closed socket that is in a TIME_WAIT state. The SO_REUSEADDR flag can be used to override this behaviour.

The SO_REUSEPORT flag is (or was) not universally available on all operating systems.

On Windows when a user issues the SO_REUSEADDR ioctl, it automatically sets the SO_REUSEPORT flag on the socket which has given rise to threads like this https://github.com/mono/mono/pull/4104, this https://github.com/dotnet/corefx/pull/11509 and commits like this https://github.com/mono/mono/commit/9d3d44013f5ca5311404737bb384b76654d414ff, that last one broke things.

The situation was further clouded by the Windows-only ExclusiveAddressUse ioctl, and this control code became conflated as a fix for the TIME_WAIT issue, for the corefx team anyway.

I've managed to identify the problem. If you compile my attached test program and run it using strace you can see the setsockopt system calls which get made by each version of Mono.

$ mcs socketTest.cs
$ strace -o socketTest.log mono socketTest.exe
$ grep setsockopt socket.log

Under Mono 4.2.1
================

  setsockopt(3, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0
  setsockopt(5, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEPORT, [1], 4) = 0

Under Mono 4.2.2
================

  setsockopt(3, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0

Under Mono 4.8.1
================

  setsockopt(3, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(3, SOL_TCP, TCP_NODELAY, [0], 4) = 0
  setsockopt(4, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(4, SOL_TCP, TCP_NODELAY, [0], 4) = 0
  setsockopt(5, SOL_IP, IP_MTU_DISCOVER, [0], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
  setsockopt(5, SOL_TCP, TCP_NODELAY, [0], 4) = 0


As you can see, the system calls to setsockopts() are quite different depending on which version of Mono we're executing under given the same C#. The breaking change was introduced between version 4.2.1 and 4.2.2.

In version 4.2.1 when the SocketOptionName.ReuseAddress flag is passed to SetSocketOption() it results in system calls which correctly set BOTH the SO_REUSEADDR and SO_REUSEPORT flags, emulating the behaviour of Windows.

In later versions, only the SO_REUSEADDR is set. Unfortunately there appears to be no way to independently set the SO_REUSEPORT flag using managed code, which has the effect of disabling this functionality.

Notice that in /mono/metadata/w32socket-unix.c at line #862 the code still exists to handle this situation correctly https://github.com/mono/mono/blob/17aa73d0ae216529b59d5aa3d0ee68007bb035c4/mono/metadata/w32socket-unix.c#L862

I'm not familiar enough with the Mono project to understand why this isn't used, but it seems instead that the method ves_icall_System_Net_Sockets_Socket_SetSocketOption_internal() in w32socket.c is responsible for setting socket options and failing to set the SO_REUSEPORT flag.

I think the possible fixes are either:

  1. Restore earlier functionality. Copy Windows and set SO_REUSEPORT in conjunction with SO_REUSEADDR.
  2. Provide a means for the SO_REUSEPORT flag to be set.

I've been unable to track down the exact commit which broke this functionality, but it would be good to get this resolved.
Comment 1 H3V2 2017-04-24 13:46:12 UTC
As a work-around, call setsockopt() with the SO_REUSEPORT option on an existing socket.

...

public static bool IsMono => Type.GetType("Mono.Runtime") != null;
public static int SOL_SOCKET = 1;
public static int SO_REUSEPORT = 15;

[DllImport("libc", SetLastError = true)]
public static extern int setsockopt(int sockfd, int level, int option, ref int value, int len);

...

var socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);

...

if (IsMono)
{
  var enable = 1;
  if (Libc.setsockopt(socket.Handle.ToInt32(), Libc.SOL_SOCKET, Libc.SO_REUSEPORT, ref enable, sizeof(int)) < 0)
  {
    throw new Exception(Marshal.GetLastWin32Error());
  }
}

Note You need to log in before you can comment on or make changes to this bug.