Bug 33254 - System.Net.HttpListenerResponse.Abort crashes server under load with unhandled System.Net.Sockets.SocketException: The socket has been shut down
Summary: System.Net.HttpListenerResponse.Abort crashes server under load with unhandle...
Status: NEW
Alias: None
Product: Class Libraries
Classification: Mono
Component: System (show other bugs)
Version: master
Hardware: Other Linux
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2015-08-20 08:59 UTC by Paul Blair
Modified: 2016-04-16 09:30 UTC (History)
4 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments

Description Paul Blair 2015-08-20 08:59:14 UTC
Mono 4.3.0 snapshot 20150611214718
Ubuntu with Linux kernel 3.19.0.26-lowlatency on ec2

Putting a simple OWIN middleware under load results in the following exception:

Unhandled Exception:
System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: The socket has been shut down
  at System.Net.Sockets.Socket.Send (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) <0x40768220 + 0x000fe> in <filename unknown>:0 
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) <0x40767fd0 + 0x000ad> in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) <0x40767fd0 + 0x00137> in <filename unknown>:0 
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) <0x40767f20 + 0x0009a> in <filename unknown>:0 
  at System.Net.ResponseStream.Close () <0x40768ab0 + 0x00166> in <filename unknown>:0 
  at System.Net.HttpConnection.Close (Boolean force_close) <0x407687a0 + 0x00054> in <filename unknown>:0 
  at System.Net.HttpListenerResponse.Close (Boolean force) <0x40768750 + 0x0003a> in <filename unknown>:0 
  at System.Net.HttpListenerResponse.Abort () <0x40769210 + 0x00028> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse.End () <0x40769090 + 0x00126> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End () <0x40768fe0 + 0x00079> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End (System.Exception ex) <0x4076a2c0 + 0x000bd> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestAsync>d__5.MoveNext () <0x40716ad0 + 0x00792> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f2650a46720 + 0x00029> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7f2650a44700 + 0x000b3> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7f2650a44660 + 0x00093> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7f2650a44610 + 0x0003a> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x7f2650a445f0 + 0x00012> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestsAsync>d__0.MoveNext () <0x406f1380 + 0x0073e> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f2650a46720 + 0x00029> in <filename unknown>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>m__1 (System.Object state) <0x7f2650a43f00 + 0x00041> in <filename unknown>:0 
  at System.Runtime.Remoting.Messaging.AsyncResult.WaitCallback_Context (System.Object state) <0x7f2650c10b40 + 0x00058> in <filename unknown>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f2650af3240 + 0x001c6> in <filename unknown>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f2650af3210 + 0x00020> in <filename unknown>:0 
  at System.Runtime.Remoting.Messaging.AsyncResult.<AsyncResult>m__0 (System.Object ) <0x7f2650c11120 + 0x00022> in <filename unknown>:0 


Steps to reproduce

1. Create a console application with the following F# code:

module Owin.Test

open System
open System.Collections.Generic
open System.Threading.Tasks
open System.IO

open Owin
open Microsoft.Owin
open Microsoft.Owin.Hosting

let middleware next env = (new OwinContext(env)).Response.WriteAsync("Hello world!")

type Startup() =
    member this.Configuration(appBuilder: IAppBuilder) = 
        appBuilder.Use(middleware) |> ignore

[<EntryPoint>]
let main argv = 
    let urlToListenOn = "http://*:8085/"
    let server = WebApp.Start<Startup>(urlToListenOn)
    stdout.WriteLine("Serving {0}, press Enter to stop.", urlToListenOn)
    stdin.ReadLine() |> ignore
    server.Dispose()
    0 


2. Libraries to include:

Owin 1.0
Microsoft.Owin 3.0.1
Microsoft.Owin.Host.HttpListener 3.0.1
Microsoft.Owin.Hosting 3.0.1

3. The exception appeared after about 30 seconds of load with 1024 connections on 2 threads, generating about 100,000 requests. Since the exception is unhandled, the middleware server crashes.
Comment 1 Paul Blair 2015-08-20 09:00:30 UTC
One more thing: I believe the exception occurred when the load was being shut off, but that's not entirely clear.
Comment 2 Paul Blair 2015-08-20 23:55:30 UTC
Benchmarking tool used for generating load is wrk, from here: https://github.com/wg/wrk

Command: 

wrk -H 'Host: localhost' -H 'Accept: application/json,text/html;q=0.9,application/xhtml+xml;q=0.9,application/xml;q=0.8,*/*;q=0.7' -H 'Connection: keep-alive' --latency -d 30 -c 1024 --timeout 1024 -t 2 http://54.147.229.26:8085/
Comment 3 Paul Blair 2015-08-20 23:57:28 UTC
I have also confirmed that the same application doesn't fall over when running on Windows with 16384 connections on 2 threads.
Comment 4 Alexander Mezhov 2015-12-09 07:22:00 UTC
Hello!

Unfortunately, I have the same situation.
Below I provided information about my code and runtime environment.

Given:
* Simple Katana OWIN SelfHost app

When:
* Client sends requests

Then:
* Sometimes Server gets 'SocketException: The socket has been shut down'

I detected that this exception occurs when Client disposes its soket and does not wait response. The worst thing is Sever does not dispose socket descriptors. It is easy to check with 'lsof' command if execute it after each expeption on Server:

> lsof -p <PID>
...
mono    41347 root    7u  sock    0,7      0t0  708962 can't identify protocol
mono    41347 root    8u  sock    0,7      0t0  709762 can't identify protocol
mono    41347 root    9u  sock    0,7      0t0  709763 can't identify protocol
mono    41347 root   10u  sock    0,7      0t0  709764 can't identify protocol
...

Number of lost descriptors increments after every client disconnection!

> uname -a

Linux ubuntu 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

> lsb_release -a

No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 14.04.3 LTS
Release:	14.04
Codename:	trusty

> mono --version

Mono JIT compiler version 4.2.1 (Stable 4.2.1.102/6dd2d0d Thu Nov 12 09:52:44 UTC 2015)
Copyright (C) 2002-2014 Novell, Inc, Xamarin Inc and Contributors. www.mono-project.com
	TLS:           __thread
	SIGSEGV:       altstack
	Notifications: epoll
	Architecture:  amd64
	Disabled:      none
	Misc:          softdebug 
	LLVM:          supported, not enabled.
	GC:            sgen

> Program.cs

using System;
using System.Text;
using System.Threading.Tasks;

using Microsoft.Owin;
using Microsoft.Owin.Hosting;

using Owin;

namespace OwinTestServer
{
    class Program
    {
        static void Main()
        {
            using (WebApp.Start<Startup>("http://+:12345"))
            {
                Console.ReadLine();
            }
        }
    }

    internal class Startup
    {
        public void Configuration(IAppBuilder app)
        {
            app.Use<TestOwinMiddleware>();
        }
    }

    internal class TestOwinMiddleware : OwinMiddleware
    {
        public TestOwinMiddleware(OwinMiddleware next) : base(next)
        {
        }

        public override Task Invoke(IOwinContext context)
        {
            context.Response.StatusCode = 200;
            context.Response.ContentType = "application/json";
            var data = Encoding.UTF8.GetBytes("{ \"message\": \"Hello!\" }");
            context.Response.Body.Write(data, 0, data.Length);
            return Task.FromResult<object>(null);
        }
    }
}

> packages.config

<?xml version="1.0" encoding="utf-8"?>
<packages>
  <package id="Microsoft.Owin" version="3.0.1" targetFramework="net452" />
  <package id="Microsoft.Owin.Host.HttpListener" version="3.0.1" targetFramework="net452" />
  <package id="Microsoft.Owin.Hosting" version="3.0.1" targetFramework="net452" />
  <package id="Microsoft.Owin.SelfHost" version="3.0.1" targetFramework="net452" />
  <package id="Owin" version="1.0" targetFramework="net452" />
</packages>

> Exception

System.IO.IOException: Write failure ---> System.Net.Sockets.SocketException: The socket has been shut down
  at System.Net.Sockets.Socket.Send (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) <0x412e5940 + 0x000e3> in <filename unknown>:0 
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) <0x412e5750 + 0x000a7> in <filename unknown>:0 
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) <0x412e5750 + 0x00123> in <filename unknown>:0 
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) <0x412e56a0 + 0x0008f> in <filename unknown>:0 
  at System.Net.ResponseStream.Close () <0x412e6440 + 0x00147> in <filename unknown>:0 
  at System.Net.HttpConnection.Close (Boolean force_close) <0x412e6190 + 0x0004f> in <filename unknown>:0 
  at System.Net.HttpListenerResponse.Close (Boolean force) <0x412e6150 + 0x00033> in <filename unknown>:0 
  at System.Net.HttpListenerResponse.Abort () <0x412e6b80 + 0x00023> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerResponse.End () <0x412e6a30 + 0x000ff> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End () <0x412e6980 + 0x00067> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.RequestProcessing.OwinHttpListenerContext.End (System.Exception ex) <0x412e74f0 + 0x0009f> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestAsync>d__5.MoveNext () <0x412d35f0 + 0x00757> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f80338016d0 + 0x00029> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x7f80337ff6b0 + 0x000a7> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x7f80337ff630 + 0x0006b> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x7f80337ff5e0 + 0x0003a> in <filename unknown>:0 
  at System.Runtime.CompilerServices.TaskAwaiter.GetResult () <0x7f80337ff5c0 + 0x00012> in <filename unknown>:0 
  at Microsoft.Owin.Host.HttpListener.OwinHttpListener+<ProcessRequestsAsync>d__0.MoveNext () <0x412b1e60 + 0x00700> in <filename unknown>:0 
--- End of stack trace from previous location where exception was thrown ---
  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x7f80338016d0 + 0x00029> in <filename unknown>:0 
  at System.Runtime.CompilerServices.AsyncMethodBuilderCore.<ThrowAsync>m__1 (System.Object state) <0x7f80337feee0 + 0x00041> in <filename unknown>:0 
  at System.Threading.QueueUserWorkItemCallback.WaitCallback_Context (System.Object state) <0x7f80338b5720 + 0x00048> in <filename unknown>:0 
  at System.Threading.ExecutionContext.RunInternal (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f80338af850 + 0x0016e> in <filename unknown>:0 
  at System.Threading.ExecutionContext.Run (System.Threading.ExecutionContext executionContext, System.Threading.ContextCallback callback, System.Object state, Boolean preserveSyncCtx) <0x7f80338af820 + 0x00020> in <filename unknown>:0 
  at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem () <0x7f80338b56b0 + 0x00053> in <filename unknown>:0 
  at System.Threading.ThreadPoolWorkQueue.Dispatch () <0x7f80338b3cc0 + 0x001d6> in <filename unknown>:0 
  at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback () <0x7f80338b5530 + 0x00008> in <filename unknown>:0
Comment 5 Alexander Mezhov 2015-12-10 04:24:44 UTC
I think, I resolved this problem. 

I set HttpListener.IgnoreWriteExceptions property to 'true' in OWIN configuration method and then register middlewares. For example:
internal class Startup
{
    public void Configuration(IAppBuilder app)
    {
        object httpListener;

        if (app.Properties.TryGetValue(typeof(HttpListener).FullName, out httpListener) && httpListener is HttpListener)
        {
            // HttpListener should not return exceptions that occur when sending the response to the client
            ((HttpListener)httpListener).IgnoreWriteExceptions = true;
        }

        app.Use<TestOwinMiddleware>();
    }
}
After that I haven't got 'SocketException: The socket has been shut down' and 'lsof' command shows that all is OK. How do you think that solution has any side-effect?

P.s. See also my conversation on Katana Project:

https://katanaproject.codeplex.com/workitem/438

Note You need to log in before you can comment on or make changes to this bug.