Bug 5926 - Several crashes in WCF server on client disconnect. See Stacktraces included.
Summary: Several crashes in WCF server on client disconnect. See Stacktraces included.
Status: NEEDINFO
Alias: None
Product: Class Libraries
Classification: Mono
Component: WCF assemblies (show other bugs)
Version: 2.10.x
Hardware: PC Linux
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2012-06-29 10:06 UTC by Robert Wilkens
Modified: 2017-09-06 16:54 UTC (History)
10 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Test case, server and client (58.06 KB, application/x-compressed-tar)
2012-06-29 10:06 UTC, Robert Wilkens
Details
Repeating called version of TestWCFServer/Client (58.52 KB, application/x-compressed-tar)
2012-07-02 09:53 UTC, Robert Wilkens
Details

Description Robert Wilkens 2012-06-29 10:06:12 UTC
I'm attaching a modified version of code that a user named Shahbour posted to the mailing list.  I extended it so that it sends more data.  The following crash occurred when I started the server on Linux, then started the client on the same machine/os (connecting to localhost), waited for the client to start sending data to server (once it built a really long string to send) then I killed the client (Ctrl+c), and the server crashed.  This is very much a lucky timing issue.  Note that what Shahbour reported on Mac it appeared to give a SocketException, while what I get here is an XmlException, when, perhaps it should have been a socket exception - it depends how the operating system handles the disconnect i guess.  The XmlException, i believe, is because we're ending the connection early, before all the data has been sent.  It is understandable for that to fail, but it shouldn't CRASH the server application.

Here is the crash message:

$ mono TestWCFServer.exe
Press enter to exit

Unhandled Exception: System.Xml.XmlException: unexpected end of file. Current depth is 4  Line 1, position 53182462.
  at Mono.Xml2.XmlTextReader.ReadContent () [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryWriter.WriteTextNode (System.Xml.XmlDictionaryReader reader, Boolean isAttribute) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryWriter.WriteNode (System.Xml.XmlDictionaryReader reader, Boolean defattr) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryWriter.WriteNode (System.Xml.XmlDictionaryReader reader, Boolean defattr) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.XmlReaderBodyWriter.OnCreateBufferedCopy (Int32 maxBufferSize) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.BodyWriter.CreateBufferedCopy (Int32 maxBufferSize) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.DefaultMessageBuffer.CreateMessage () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Logger.LogMessage (MessageLogSourceKind sourceKind, System.ServiceModel.Channels.Message& msg, Int64 maxMessageSize) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__1B (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0
Comment 1 Robert Wilkens 2012-06-29 10:06:41 UTC
Created attachment 2133 [details]
Test case, server and client
Comment 2 Robert Wilkens 2012-06-29 10:25:58 UTC
Please note this sample code may slow down your PC for up to a minute or so (maybe longer) while it does string building.  If you want to modify the source you can probably significanlty reduce the string sizes/loops-on-creating-string counts -- in fact i'd highly recommend it.
Comment 3 Robert Wilkens 2012-06-29 10:35:31 UTC
I reproduced the same basic error on mac too.... Basic step: Run, wait till it tried to connect, wait an additional 4-5 seconds, Press Ctrl+C on client, Server crashes.

$ mono TestWCFServer.exe
Press enter to exit
Exception unexpected end of file. Current depth is 4  Line 1, position 21917750.   at Mono.Xml2.XmlTextReader.ReadContent () [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.ReadContentString (Boolean isText) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.ReadContentString () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.ReadContentAsString () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryReader.ReadContentAsString (Int32 maxStringContentLength) [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryReader.ReadContentAsString () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryReader.ReadElementContentAsString () [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.DeserializePrimitive (System.Type type, System.Xml.XmlReader reader, System.Xml.XmlQualifiedName qname) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Type type, System.Xml.XmlReader reader) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Xml.XmlReader reader, System.Type declaredType, System.Runtime.Serialization.KnownTypeCollection knownTypes, IDataContractSurrogate surrogate, System.Runtime.Serialization.DataContractResolver resolver, System.Runtime.Serialization.DataContractResolver defaultResolver, System.String name, System.String ns, Boolean verifyObjectName) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.DataContractSerializer.ReadObject (System.Xml.XmlDictionaryReader reader, Boolean verifyObjectName) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlObjectSerializer.ReadObject (System.Xml.XmlDictionaryReader reader) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.ReadMessagePart (System.ServiceModel.Description.MessagePartDescription part, System.Xml.XmlDictionaryReader r) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.MessageToParts (System.ServiceModel.Description.MessageDescription md, System.ServiceModel.Channels.Message message) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseMessagesFormatter.DeserializeRequest (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeRequest (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.BuildInvokeParams (System.ServiceModel.Dispatcher.MessageProcessingContext mrc, System.Object[]& parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.DoProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.HandlersChain.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessor.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
Comment 4 shahbour 2012-06-29 10:48:02 UTC
Hello

As Rob said the above crash happen more often when the there is lot of data exchanged and the error change depending when the client closes his application/browser .

If it is closed while sending we get usually XMLRead error and if after that we most probably got Connection reset by peer.

The bug is that the hosting application is crashing .

Below are some more Stack info to light on the issue

[0x7f0b42f55700:] EXCEPTION handling: System.Net.Sockets.SocketException: Connection reset by peer
[0x7f0b42f55700:] EXCEPTION handling: System.IO.IOException: Read failure
[0x7f0b42f55700:] EXCEPTION handling: System.IO.IOException: Read failure

Unhandled Exception: System.IO.IOException: Read failure ---> System.Net.Sockets.SocketException: Connection reset by peer
  at System.Net.Sockets.Socket.Receive (System.Byte[] buffer, Int32 offset, Int32 size, SocketFlags flags) [0x000ad] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net.Sockets/Socket.cs:1606
  at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, Int32 offset, Int32 size) [0x00067] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net.Sockets/NetworkStream.cs:399
  --- End of inner exception stack trace ---
  at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, Int32 offset, Int32 size) [0x00078] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net.Sockets/NetworkStream.cs:401
  at System.Net.RequestStream.Read (System.Byte[] buffer, Int32 offset, Int32 count) [0x0003c] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net/RequestStream.cs:142
  at System.IO.StreamReader.ReadBuffer () [0x00012] in /usr/local/src/mono-2.11.1/mcs/class/corlib/System.IO/StreamReader.cs:394
  at System.IO.StreamReader.Read (System.Char[] buffer, Int32 index, Int32 count) [0x00062] in /usr/local/src/mono-2.11.1/mcs/class/corlib/System.IO/StreamReader.cs:471
  at Mono.Xml2.XmlTextReader.ReadTextReader (Int32 remained) [0x0004f] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1256
  at Mono.Xml2.XmlTextReader.PeekChar () [0x00047] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1210
  at Mono.Xml2.XmlTextReader.ReadContent () [0x0003f] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlTextReader.cs:1310
  at Mono.Xml2.XmlTextReader.Read () [0x00141] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlTextReader.cs:626
  at System.Xml.XmlTextReader.Read () [0x0006b] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlTextReader2.cs:564
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in /usr/local/src/mono-2.11.1/mcs/class/System.Runtime.Serialization/System.Xml/XmlSimpleDictionaryReader.cs:253
  at System.Xml.XmlReader.MoveToContent () [0x00050] in /usr/local/src/mono-2.11.1/mcs/class/System.XML/System.Xml/XmlReader.cs:574
  at System.ServiceModel.Channels.XmlReaderMessage.ReadEnvelopeStart () [0x00000] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/MessageImpl.cs:146
  at System.ServiceModel.Channels.XmlReaderMessage..ctor (System.ServiceModel.Channels.MessageVersion version, System.Xml.XmlDictionaryReader reader, Int32 maxSizeOfHeaders) [0x00026] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/MessageImpl.cs:70
  at System.ServiceModel.Channels.Message.CreateMessage (System.Xml.XmlDictionaryReader envelopeReader, Int32 maxSizeOfHeaders, System.ServiceModel.Channels.MessageVersion version) [0x00022] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/Message.cs:427
  at System.ServiceModel.Channels.TextMessageEncoder.ReadMessage (System.IO.Stream stream, Int32 maxSizeOfHeaders, System.String contentType) [0x00011] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/TextMessageEncoder.cs:87
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00081] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpReplyChannel.cs:229
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x000aa] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpReplyChannel.cs:172
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__10 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x0002f] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/ReplyChannelBase.cs:126 

Another Example

[0x43b74940:] EXCEPTION handling: System.Net.Sockets.SocketException: The socket has been shut down
[0x43b74940:] EXCEPTION handling: System.IO.IOException: Write failure
Exception Write failure   at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x0008e] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net.Sockets/NetworkStream.cs:445
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00029] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net/ResponseStream.cs:129
  at System.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x000dd] in /usr/local/src/mono-2.11.1/mcs/class/System/System.Net/ResponseStream.cs:162
  at System.ServiceModel.Channels.Http.HttpRequestContext.InternalReply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00157] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpRequestContext.cs:160
  at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpRequestContext.cs:101
  at System.ServiceModel.Dispatcher.MessageProcessingContext.Reply (Boolean useTimeout) [0x00026] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/MessageProcessingContext.cs:96
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.Reply (System.ServiceModel.Dispatcher.MessageProcessingContext mrc, Boolean useTimeout) [0x0001d] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/OperationInvokerHandler.cs:69
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00044] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/OperationInvokerHandler.cs:29
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseRequestProcessorHandler.cs:15
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00017] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseRequestProcessorHandler.cs:16
  at System.ServiceModel.Dispatcher.HandlersChain.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x0000b] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseRequestProcessor.cs:72
  at System.ServiceModel.Dispatcher.BaseRequestProcessor.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00018] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/BaseRequestProcessor.cs:26 

Another Example

System.InvalidOperationException: Cannot be changed after headers are sent. at System.Net.HttpListenerResponse.set_ContentType (System.String value) [0x00027] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System/System.Net/HttpListenerResponse.cs:107 at System.ServiceModel.Channels.Http.HttpStandaloneResponseInfo.set_ContentType (System.String value) [0x00000] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpContextInfo.cs:274 at System.ServiceModel.Channels.Http.HttpRequestContext.InternalReply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00046] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpRequestContext.cs:140 at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpRequestContext.cs:101 at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg) [0x00000] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpRequestContext.cs:96 at System.ServiceModel.Dispatcher.ListenerLoopManager.ProcessRequest (IReplyChannel reply, System.ServiceModel.Channels.RequestContext rc) [0x0003b] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:601 at System.ServiceModel.Dispatcher.ListenerLoopManager.TryReceiveRequestDone (IAsyncResult result) [0x0001a] in /private/tmp/monobuild/build/BUILD/mono-2.10.9/mcs/class/System.ServiceModel/System.ServiceModel.Dispatcher/ChannelDispatcher.cs:575
Comment 5 Robert Wilkens 2012-06-29 10:53:02 UTC
On same test code, on linux, after playing with string length transmitted on client and server a bit, I received yet another stack trace on server which follows.

Unhandled Exception: System.Net.Sockets.SocketException: The socket is not connected
  at System.Net.Sockets.Socket.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.Net.HttpConnection.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.Net.HttpListenerRequest.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpStandaloneRequestInfo.get_ClientIPAddress () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__1B (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0
Comment 6 shahbour 2012-07-01 19:12:00 UTC
Hello

Another Stack Crash

[0x7f4a06eee700:] EXCEPTION handling: System.ArgumentOutOfRangeException: Argument is out of range.
Parameter name: timeout
[0x7f4a06eee700:] EXCEPTION handling: System.ArgumentOutOfRangeException: Argument is out of range.
Parameter name: timeout

Unhandled Exception: System.ArgumentOutOfRangeException: Argument is out of range.
Parameter name: timeout
  at System.Threading.WaitHandle.WaitOne (TimeSpan timeout, Boolean exitContext) [0x00073] in /usr/local/src/mono-2.11.1/mcs/class/corlib/System.Threading/WaitHandle.cs:427
  at System.Threading.WaitHandle.WaitOne (TimeSpan timeout) [0x00000] in /usr/local/src/mono-2.11.1/mcs/class/corlib/System.Threading/WaitHandle.cs:401
  at System.ServiceModel.Channels.Http.HttpListenerManager.TryDequeueRequest (System.ServiceModel.Dispatcher.ChannelDispatcher channel, TimeSpan timeout, System.ServiceModel.Channels.Http.HttpContextInfo& context) [0x00057] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpListenerManager.cs:99
  at System.ServiceModel.Channels.Http.HttpListenerManager.TryDequeueRequest (System.ServiceModel.Dispatcher.ChannelDispatcher channel, TimeSpan timeout, System.ServiceModel.Channels.Http.HttpContextInfo& context) [0x00065] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpListenerManager.cs:100
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00003] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels.Http/HttpReplyChannel.cs:151
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__10 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x0002f] in /usr/local/src/mono-2.11.1/mcs/class/System.ServiceModel/System.ServiceModel.Channels/ReplyChannelBase.cs:126
Comment 7 Robert Wilkens 2012-07-01 20:11:14 UTC
IF I understand comment 6's stack trace.. 

The timeout calculation:

timeout - (DateTime.Now - start)

around line 99-100 of HttpListenerManager...

could result in a NEGATIVE timeout value, which may explain that crash.  I have a "workaround" for that one, which looks something like this, but this code isn't tested yet, note I added TimeOutRevised and related code, the 'am lazy' part is not my code, not that i'm not lazy:

                                if (q.Count == 0) {
                                        bool ret = ce.WaitHandle.WaitOne (timeout);
                                        int TimeOutRevised = timeout - (DateTime.Now - start);
                                        if (TimeOutRevised > 0) {
                                                return ret && TryDequeueRequest (channel, TimeOutRevised, out context); // recurse, am lazy :/
                                        } else {
                                                return ret;
                                        }
                                }

I'm trying to build mono from scratch now (had to do a make clean after pulling in latest updates and couldn't build).  Will try to test this in with some of the other changes i've made.
Comment 8 Robert Wilkens 2012-07-02 08:06:58 UTC
I am still testing and hit another one...

It's another related stack trace, similar to others in that it happenned on XmlTextReader reading, but it was a different exception


Exception Read failure   at System.Net.Sockets.NetworkStream.Read (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0 
  at System.Net.RequestStream.Read (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0 
  at System.IO.StreamReader.ReadBuffer () [0x00000] in <filename unknown>:0 
  at System.IO.StreamReader.Read (System.Char[] buffer, Int32 index, Int32 count) [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.ReadTextReader (Int32 remained) [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.PeekChar () [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.ReadText (Boolean notWhitespace) [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.ReadContent () [0x00000] in <filename unknown>:0 
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.ReadStartElement () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlDictionaryReader.ReadElementContentAsString () [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.DeserializePrimitive (System.Type type, System.Xml.XmlReader reader, System.Xml.XmlQualifiedName qname) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Type type, System.Xml.XmlReader reader) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlFormatterDeserializer.Deserialize (System.Xml.XmlReader reader, System.Type declaredType, System.Runtime.Serialization.KnownTypeCollection knownTypes, IDataContractSurrogate surrogate, System.Runtime.Serialization.DataContractResolver resolver, System.Runtime.Serialization.DataContractResolver defaultResolver, System.String name, System.String ns, Boolean verifyObjectName) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.DataContractSerializer.ReadObject (System.Xml.XmlDictionaryReader reader, Boolean verifyObjectName) [0x00000] in <filename unknown>:0 
  at System.Runtime.Serialization.XmlObjectSerializer.ReadObject (System.Xml.XmlDictionaryReader reader) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.ReadMessagePart (System.ServiceModel.Description.MessagePartDescription part, System.Xml.XmlDictionaryReader r) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.DataContractMessagesFormatter.MessageToParts (System.ServiceModel.Description.MessageDescription md, System.ServiceModel.Channels.Message message) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseMessagesFormatter.DeserializeRequest (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationFormatter.DeserializeRequest (System.ServiceModel.Channels.Message message, System.Object[] parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.BuildInvokeParams (System.ServiceModel.Dispatcher.MessageProcessingContext mrc, System.Object[]& parameters) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.DoProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.HandlersChain.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Dispatcher.BaseRequestProcessor.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
Comment 9 Robert Wilkens 2012-07-02 08:10:21 UTC
That last stack trace _might_ not have caused a crash though, but it still is worth noting that we should probably handle a dropped connection more clean than dumping a stack trace.
Comment 10 Robert Wilkens 2012-07-02 08:17:00 UTC
In terms of catching exceptions, Comment 8 may not be clear what kind of exception it is, but from looking at code it's an IOException.
Comment 11 Robert Wilkens 2012-07-02 09:23:57 UTC
Comment 1 + Comment 3 (the ones i reported) may not have resulted in crashes.. These are the ones which fail in XmlTextReadear with a 'depth' message and something like unexpected end of file...

I've got a working fix for that which basically doesn't print a whole stacktrace and instead prints something like this:


XmlException (Dropped Connection?): unexpected end of file. Current depth is 4  Line 1, position 3866622.
RDWDEBUG: Condition 1 met - SocketException caught
XmlException (Dropped Connection?): unexpected end of file. Current depth is 4  Line 1, position 1622014.

PLease note: Condition 1 met was also my fix (perhaps included in combined.patch on mailing list earlier) where i was testing to see if that would ever be hit.  That would previously result in a crash.  Condition 1, specifically was where we were in TryReceiveRequest and actually were crashing entirely.  This was specifically Commennt 4 Stacktrace 1 which was being hit.  I modified the message better than "Condition 1 met" now that i know it is indeed hit - but note that we didn't crash, it continued.

There is one other fix i have queued was where we were in ProcessRequest in ChannelDispatcher and failed in Reply.  I haven't been able to reproduce that one (which i call condition 2.  I believe this was one of the stacktrace's in comment #4.  When I can reproduce that much, perhaps i will submit a pull request with all of these fixes so far in it, and if it's pulled/merged you can try to reproduce these problems again and see what's left.
Comment 12 Robert Wilkens 2012-07-02 09:37:37 UTC
Another stacktrace, i think i have a fix for it already in place too now, but to document it (this one caused a crash):
Unhandled Exception: System.Xml.XmlException: Document element did not appear.  Line 1, position 1.
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.MoveToContent () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.XmlReaderMessage.ReadEnvelopeStart () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.XmlReaderMessage..ctor (System.ServiceModel.Channels.MessageVersion version, System.Xml.XmlDictionaryReader reader, Int32 maxSizeOfHeaders) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Message.CreateMessage (System.Xml.XmlDictionaryReader envelopeReader, Int32 maxSizeOfHeaders, System.ServiceModel.Channels.MessageVersion version) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.TextMessageEncoder.ReadMessage (System.IO.Stream stream, Int32 maxSizeOfHeaders, System.String contentType) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__21 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0
Comment 13 Robert Wilkens 2012-07-02 09:53:46 UTC
Created attachment 2146 [details]
Repeating called version of TestWCFServer/Client

This revised version of the test case will repeatedly call the server from the client (first rnu TestWCFServer, then run in another window TestWCFClient).  To test Press Ctrl+C to break the client at various spots, and with luck if you keep trying one of the attached stacktraces comes up.  This is not 100% my code, this was provided by Shahbour, but i modified it so that it sends a big chunk of data and calls the server repeatedly.
Comment 14 Robert Wilkens 2012-07-02 10:42:23 UTC
Submitted a pull request with the fixes i had so far:
https://github.com/mono/mono/pull/368
Comment 15 Robert Wilkens 2012-07-04 09:53:05 UTC
Re: The pull request.  Shahbour: I sent you the patches separately as you requested on Monday (about 2 days ago).  Did you have any luck with them?  Please comment here in the bug report so i know whether or not more needs to be done.  There is no rush on this, so if you're busy or (if in the u.s.a.) enjoying a holiday today, no reason to hurry.  I'm holding off working on other bugs until this is resolved.  I don't necessarily need to wait until the pull request is accepted or rejected, but i do want to wait until i know if this is working better for you.
Comment 16 Robert Wilkens 2012-07-06 21:10:42 UTC
The pull request was revised into one commit here :

https://github.com/mono/mono/pull/377

I didn't realize the original pull request had merges in them which should not have been there.  I'm new to git.
Comment 17 shahbour 2012-07-09 07:22:22 UTC
Hello

Sorry i was not able to reply before i got very busy on another project.

Now i applied the patched you have sent me on one of my nodes that is crashing frequently and started my application.

I will keep it under monitoring and will update you in like two days.

BR
Shahbour
Comment 18 Robert Wilkens 2012-07-09 08:06:57 UTC
Hoping for the best, but if you have another unexpected stack trace related to this problem, even if it's the same as one above, please list it here again.  Today i will not be able to dedicate any time to working on mono, due to family health issues, tomorrow or the next day not sure either.
Comment 19 Robert Wilkens 2012-07-11 07:48:31 UTC
The pull request was updated with the following:

-Where I was handling IOException, SocketException is now also being handled.
-Where I was handling SocketExceptoin, IOException is now also being handled.

This catches at least one additional crash which i hadn't previously handled, which Shahbour found.
Comment 20 u.hertlein 2012-12-17 22:50:37 UTC
FWIW I'm currently seeing similar errors with Mono-3.0.0 on Linux:

System.Net.Sockets.SocketException: The socket is not connected
  at System.Net.Sockets.Socket.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.Net.HttpConnection.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.Net.HttpListenerRequest.get_RemoteEndPoint () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpStandaloneRequestInfo.get_ClientIPAddress () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__21 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0

and

System.Xml.XmlException: Document element did not appear.  Line 1, position 1.
  at Mono.Xml2.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlTextReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlSimpleDictionaryReader.Read () [0x00000] in <filename unknown>:0 
  at System.Xml.XmlReader.MoveToContent () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.XmlReaderMessage.ReadEnvelopeStart () [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.XmlReaderMessage..ctor (System.ServiceModel.Channels.MessageVersion version, System.Xml.XmlDictionaryReader reader, Int32 maxSizeOfHeaders) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Message.CreateMessage (System.Xml.XmlDictionaryReader envelopeReader, Int32 maxSizeOfHeaders, System.ServiceModel.Channels.MessageVersion version) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.TextMessageEncoder.ReadMessage (System.IO.Stream stream, Int32 maxSizeOfHeaders, System.String contentType) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0 
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__21 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0 

Oddly enough the stack trace stops there so I cannot easily spot where this is coming from.  But from a cursory glance there isn't any exception handling from 'BeginTryReceiveRequest' downwards.
Comment 21 jacotb 2014-01-10 18:38:21 UTC
Can confirm. When a client disconnects prematurely, the runtime throws the following fatal exception:




Exception Write failure   at System.Net.Sockets.NetworkStream.Write (System.Byte[] buffer, Int32 offset, Int32 size) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.InternalWrite (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at System.Net.ResponseStream.Write (System.Byte[] buffer, Int32 offset, Int32 count) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpRequestContext.InternalReply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.MessageProcessingContext.Reply (Boolean useTimeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.Reply (System.ServiceModel.Dispatcher.MessageProcessingContext mrc, Boolean useTimeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.OperationInvokerHandler.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.BaseRequestProcessorHandler.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.HandlersChain.ProcessRequestChain (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.BaseRequestProcessor.ProcessRequest (System.ServiceModel.Dispatcher.MessageProcessingContext mrc) [0x00000] in <filename unknown>:0

Unhandled Exception:
System.InvalidOperationException: Cannot be changed after headers are sent.
  at System.Net.HttpListenerResponse.set_ContentType (System.String value) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpStandaloneResponseInfo.set_ContentType (System.String value) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpRequestContext.InternalReply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg, TimeSpan timeout) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpRequestContext.Reply (System.ServiceModel.Channels.Message msg) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.ListenerLoopManager.ProcessRequest (IReplyChannel reply, System.ServiceModel.Channels.RequestContext rc) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Dispatcher.ListenerLoopManager.TryReceiveRequestDone (IAsyncResult result) [0x00000] in <filename unknown>:0
Comment 22 jacotb 2014-01-11 10:35:59 UTC
NB: This is a bug that basically renders the whole of WCF useless for any use in any practical environment, because any client can (willingly or not) crash the entire service.

It's still present in both 3.0.6 and trunk (tested).
Comment 23 Michael Ayvazyan 2014-03-14 11:49:56 UTC
I get very similar error in the latest Mono as well on RaspberryPi.

Mono JIT compiler version 3.2.8 (tarball Thu Mar 13 08:03:10 UTC 2014)
Linux raspberrypi 3.10.33+ #654 PREEMPT Fri Mar 7 16:32:08 GMT 2014 armv6l GNU/Linux


CurrentDomainUnhandledException System.NullReferenceException: Object reference not set to an instance of an object
  at System.ServiceModel.Channels.Http.HttpStandaloneRequestInfo.get_ClientIPAddress () [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpReplyChannel.CreatePostMessage (System.ServiceModel.Channels.Http.HttpContextInfo ctxi) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.Http.HttpReplyChannel.TryReceiveRequest (TimeSpan timeout, System.ServiceModel.Channels.RequestContext& context) [0x00000] in <filename unknown>:0
  at System.ServiceModel.Channels.ReplyChannelBase.<BeginTryReceiveRequest>m__0 (TimeSpan tout, System.ServiceModel.Channels.RequestContext& ctx) [0x00000] in <filename unknown>:0
2014-03-14 15:00:23.8998 Fatal VendorsExchange.OVP.ESB.NodeConsole.Program.CurrentDomainUnhandledException IsTerminating: True
Comment 24 Tadej 2014-09-02 11:51:22 UTC
I have similar error ("Cannot be changed after headers are sent" like jacotb@gmail.com) on Beaglebone Black running latest version of Arch Linux and Mono 3.8.1 built from source.

It also looks like the error occures when a client disconnects prematurely.
Comment 25 Miguel de Icaza [MSFT] 2015-02-15 09:43:19 UTC
I posted an updated patch (untested) here:

https://gist.github.com/migueldeicaza/01aaf064b1bf626f8cc0

I would love if folks could try that out, as I am no WCF expert.

Note You need to log in before you can comment on or make changes to this bug.