Bug 44994

Summary: DeflateStream decompression is incomplete if reading byte-by-byte
Product: iOS Reporter: mfulker
Component: BCL Class LibrariesAssignee: João Matos <joao.matos>
Status: VERIFIED FIXED    
Severity: normal CC: amund, jatint, kumpera, masafa, mono-bugs+monotouch, timothy.risi
Priority: Normal    
Version: unspecified   
Target Milestone: (C9)   
Hardware: Macintosh   
OS: Mac OS   
Tags: Is this bug a regression?: ---
Last known good build:
Attachments: Working windows console app and Xamarin iOS app with premature end of stream

Description mfulker 2016-10-03 20:37:53 UTC
Created attachment 17850 [details]
Working windows console app and Xamarin iOS app with premature end of stream

When decompressing certain compressed files, reading the DeflateStream byte-by-byte with ReadByte() will omit the last few bytes, seemingly hitting end-of-stream prematurely.
On the same file, using DeflateStream.CopyTo(otherStream) copies the data correctly.

(I've also encountered a problem with EndOfStreamException when trying to use a BinaryReader on the DeflateStream for the same files.)

I'm not sure why only certain files cause it or what the pattern is in the failing files.


In the attached zip are solutions for a Xamarin iOS app and a Windows Console App.
Each has two compressed file resources that they attempt to decompress: "compressed.bin" and "compressed2.bin".
The decompressed size is calculated for each, both via CopyTo into a MemoryStream, and by reading to the end with ReadByte().
The only incorrect case is when reading "compressed.bin" with ReadByte on iOS, which yields 125377 bytes instead of 125387 bytes.
Using ReadByte on Windows or using CopyTo will yield the correct deflated size for the first file.
The other file "compressed2.bin" decompresses correctly on both platforms with both approaches.
Comment 1 mfulker 2016-10-04 21:30:56 UTC
I discovered that appending some zero-value bytes to the end of the file will fix every case of the issue I've seen.

Doing that as a workaround shouldn't cause any other output problems with the Deflate algorithm, should it?
(My knowledge of the Deflate specification is that it works on a block-by-block basis and that each block has an "is last block" header bit, so I don't expect superfluous bytes in input to cause extra bytes on output or anything like that.)

As a temporary workaround (and depending on ETA for a real fix), I might create a read-only Stream wrapper that appends a small number of superfluous bytes to the end of the wrapped FileStream (without actually modifying the file).
Comment 2 mfulker 2016-10-05 17:44:54 UTC
I actually discovered one particular (very small) compressed file also has incorrect results even with CopyTo:

data:application/octet-stream;base64,7cWxCQAgDACwpeBjgqsgXiHU0fd9QzBLErX1EQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADepcxcuU/atm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm37zy8=


The correct decompressed length should be 81942 bytes according to windows.
The decompressed length when using CopyTo on iOS is 81921.
The decompressed length when reading byte-by-byte on iOS is 81697.
Comment 3 Rolf Bjarne Kvinge [MSFT] 2016-10-06 06:16:45 UTC
This might be related to bug #34916.
Comment 4 João Matos 2016-10-27 19:37:20 UTC
https://github.com/mono/mono/pull/3844
Comment 6 Marek Safar 2016-10-28 10:48:01 UTC
@joao could you merge it to Mono 4.8 as well