Bug 58824 - Large File Upload
Summary: Large File Upload
Status: RESOLVED NOT_REPRODUCIBLE
Alias: None
Product: Class Libraries
Classification: Mono
Component: System (show other bugs)
Version: master
Hardware: Other Linux
: --- normal
Target Milestone: Untriaged
Assignee: Bugzilla
URL:
Depends on:
Blocks:
 
Reported: 2017-08-16 19:56 UTC by Paul Mackintosh
Modified: 2017-10-13 14:49 UTC (History)
5 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
execution log (1.05 MB, text/plain)
2017-08-16 21:21 UTC, Paul Mackintosh
Details
code sample (2.32 KB, text/plain)
2017-08-16 21:24 UTC, Paul Mackintosh
Details

Description Paul Mackintosh 2017-08-16 19:56:48 UTC
When uploading an MultipartFormDataContent using HttpClient.PostAsync to an https endpoint and where our parts consist of a StringContent (30 chars) followed by a StreamContent (Header 'application/zip') we see the Exception below.  This occurs when our StreamContent length exceeds ~28 MB.  Smaller file sizes succeed.  Running the same code on Windows from Visual Studio succeeds with a 120 MB stream, the same is true when running the same code in dotnetcore on the same VM as the mono application runs and fails:


System.Net.Http.HttpRequestException: An error occurred while sending the request 
---> System.IO.IOException: Error writing request 
---> System.Net.Sockets.SocketException: The socket has been shut down#011  
at System.Net.WebConnection.EndWrite (System.Net.HttpWebRequest request, System.Boolean throwOnError, System.IAsyncResult result) [0x000ad] in <5641e4edad4f4464ba58c620a7b8ea48>:0 #011
at System.Net.WebConnectionStream.WriteAsyncCB (System.IAsyncResult r) [0x00013] in <5641e4edad4f4464ba58c620a7b8ea48>:0 #011   
--- End of inner exception stack trace ---#011
at System.Net.WebConnectionStream.EndWrite (System.IAsyncResult r) [0x000d9] in <5641e4edad4f4464ba58c620a7b8ea48>:0 #011  
at System.IO.Stream.<BeginEndWriteAsync>m__8 (System.IO.Stream stream, System.IAsyncResult asyncResult) [0x00000] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at (wrapper delegate-invoke) System.Func`3[System.IO.Stream,System.IAsyncResult,System.Threading.Tasks.VoidTaskResult]:invoke_TResult_T1_T2 (System.IO.Stream,System.IAsyncResult)#011  at System.Threading.Tasks.TaskFactory`1+FromAsyncTrimPromise`1[TResult,TInstance].Complete (TInstance thisRef, System.Func`3[T1,T2,TResult] endMethod, System.IAsyncResult asyncResult, System.Boolean requiresSynchronization) [0x00002] in <f712f98eb8e445c8918edaf595bbe465>:0 #011
--- End of stack trace from previous location where exception was thrown ---#011  

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f712f98eb8e445c8918edaf595bbe465>:0 #011
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Net.Http.MultipartContent+<SerializeToStreamAsync>c__async0.MoveNext () [0x0026e] in <cbcaef35d3364f8bb9ae7449eac280e9>:0 #011
--- End of stack trace from previous location where exception was thrown ---#011  

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable+ConfiguredTaskAwaiter.GetResult () [0x00000] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x00363] in <cbcaef35d3364f8bb9ae7449eac280e9>:0 #011   
--- End of inner exception stack trace ---#011  
at System.Net.Http.HttpClientHandler+<SendAsync>c__async0.MoveNext () [0x0049a] in <cbcaef35d3364f8bb9ae7449eac280e9>:0 #011
--- End of stack trace from previous location where exception was thrown ---#011  

at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () [0x0000c] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) [0x0004e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) [0x0002e] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) [0x0000b] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () [0x00000] in <f712f98eb8e445c8918edaf595bbe465>:0 #011  
at System.Net.Http.HttpClient+<SendAsyncWorker>c__async0.MoveNext () [0x000f3] in <cbcaef35d3364f8bb9ae7449eac280e9>:0
Comment 1 Paul Mackintosh 2017-08-16 19:57:54 UTC
Forgot to add, we have the same issue on version 5
Comment 2 Ludovic Henry 2017-08-16 20:24:13 UTC
Could you please attach a log of the execution when passing the `MONO_LOG_LEVEL=debug MONO_LOG_MASK=io-layer` environment variables? Could you also please provide a reproduction case that we can easily run on our end? Thank you.
Comment 3 Paul Mackintosh 2017-08-16 21:21:48 UTC
Created attachment 24253 [details]
execution log

Log attached.
Comment 4 Paul Mackintosh 2017-08-16 21:24:11 UTC
Created attachment 24254 [details]
code sample

Attaching the code I'm running, unfortunately I'm unable to share the input output endpoints with you directly
Comment 5 Paul Mackintosh 2017-08-17 16:41:28 UTC
We asked the 3rd party we're sending our files to to give us an HTTP endpoint rather than HTTPS so that we could examine the TCP stuff more easily.

When we tested against HTTP the issue went away, so it points to an issue with SSL.
Comment 6 Ludovic Henry 2017-08-17 17:04:06 UTC
Thanks for the attached log and reproduction case, as well as the fact that it reproduces with SSL only. Can you reproduce if you are posting the string and the file to any HTTPS endpoint, like for example a local or remote (over the internet) apache or nginx server (or any other HTTPS server) that doesn't do anything with the content?
Comment 7 Paul Mackintosh 2017-08-17 17:11:58 UTC
I tried this a couple of days ago and no, the issue doesn't occur.

I set an HTTPS endpoint up on a Windows VM running on the same machine and I was able to POST the file and get a response.
Comment 8 Paul Mackintosh 2017-08-17 17:30:58 UTC
I've just tried the request with curl from the Ubuntu VM where I've been running the sample I sent you.

The curl request succeeds.  

I sent the same zipped content stream that failed in mono, i.e. I used curl to send the stream created by CreateZipStream in the sample code.
Comment 9 Paul Mackintosh 2017-08-17 17:35:10 UTC
Observation from our 3rd party:

"Yes, I see it. Appears in our logs as "Processing of multipart/form-data request failed. Unexpected EOF read on the socket". So it does appear that the data/request is being truncated/cut off and not by our side. I see the whole request took 14.251s and we returned a 400 bad request code. (We do have an idle timeout of 15s but that didn't come into play here (would have been a different code)). So yes, looks like your client is cutting out."
Comment 10 Rodrigo Kumpera 2017-10-11 18:00:24 UTC
Thanks for the test case, we'll take a look at it soon.
Comment 11 Marek Safar 2017-10-13 14:49:09 UTC
I cannot reproduce it with Mono 5.4 and using following entry points in your sample

private const string _inputFileEndpoint = @"http://mirror.filearena.net/pub/speed/SpeedTest_64MB.dat";
private const string _destinationEndpoint = @"https://posttestserver.com/post.php";

Note You need to log in before you can comment on or make changes to this bug.