Let's pretend I am developing a new client-server solution with a custom protocol, using TCP sockets.
The server part is implemented in C# and sends arbitrary messages over the same open connection. The sizes vary between few bytes and few hundreds kilobytes. The sending behavior is adapting for low latency or better throughput in case of congestion (i.e. it initially goes with plain Socket.Send if no other activity is present, or Socket.BeginSend with callback and in case there are more input chunks from other threads arrived in meantime, it tries Socket.BeginSend with a list of chunks - in the callbacks, EndSend returns the transferred size, the last chunk resp. the chunk list is then shortened from the beginning accordingly to match the number of transferred bytes).
The problem is: sometimes, when a special pattern of chunks is added to the write list, it seems that the resulting data contains garbage on the wire. The result returned from EndSend looks fine. But in Wireshark we see some data immediately after the first message part which does not belong there, it's not strictly from the second chunk either. It looks like it's some already used memory, the string correspond to what was sent before.
The pattern to trigger this bug seems to be a list for BeginSend with one big message (about 200-300kb) and a handful of small ones. I am not sure about the sizes (maybe it needs to be bigger than configured send and receive buffer?).
I have also not checked the value returned by EndSend yet, this might be a culprit too (i.e. reporting to small, making my code resend some duplicated data).
- it's not reproducible with mono 3.2.8 from Ubuntu 14 LTS (although it has its own glitches wrt TCP behavior)
- it's not reproducible on Windows
- it's almost always reproducible with mono 4.5.1 (today's Git snapshot)
- it happens in a VM as well as on bare metal
Could this be related to https://bugzilla.xamarin.com/show_bug.cgi?id=19665?
Which means data from another socket is sent to the file handle?
IMHO unlikely, from application POV there was not enough concurrency to trigger races, i.e. the test app had single-threaded communication code.
And my workaround for the mentioned problem was manual segmentation of such large chunks.
I have seen this also with 3.2.8 and also with small chuncks of data. It is hard te reproduce, still working on a way to trigger it consistently.
I don't know if we are talking about the same issue.
What I can reproduce is when sending data under high load, with also some connections are failing, the HTTP body contains a header and content of a different request than the request currently done.
In practice (production) we have also seen 'garbage' being appended, so it might be that while trying to reproduce, we've triggered another issue.
Anyway, it is always the case you send something to a socket and the receiving side is receiving different data, which is always sent by the client when you look at the wireshark logs.
I will upload our reproduction code.
The server part (WebApi.Template1) can be run on a Windows machine under IIS.
The client part (ConsoleAppplication11) must be run on a Linux box with mono, just tried 3.8.0 and it might take a few tries, but then reproduces the issue.
Upload is too big, see:
It creates a server on port /WebApi.Template1 on port 80
It looks if the number in the query string of the request is also repeated 10000 times in the body (formatted as a 2 position hexadecimal number)
If this is the case, it returns 10000 times OO
If not, it returns "FAILURE!!!"
(Run the server on Windows IIS)
The client opens a 30 threads to the working port (80) and does a POST request with the an increasing number, which is put in the query string and also sent 10000 times in the body (formatted as a 2 position hexadecimal number)
At the same time it opens 30 threads to a non-available port (8080) and tries to do the same requests (this seems to increase the odds of reproducing)
Sometimes it triggers a failure almost immediately, sometimes the client part needs to be restarted a few times to make it fail.
Last test as run on mono 3.8.0, but tested with many different versions, including 4.4.1