Bug 44994 - DeflateStream decompression is incomplete if reading byte-by-byte
Summary: DeflateStream decompression is incomplete if reading byte-by-byte
Status: VERIFIED FIXED
Alias: None
Product: iOS
Classification: Xamarin
Component: BCL Class Libraries (show other bugs)
Version: unspecified
Hardware: Macintosh Mac OS
: Normal normal
Target Milestone: (C9)
Assignee: João Matos
URL:
Depends on:
Blocks:
 
Reported: 2016-10-03 20:37 UTC by mfulker
Modified: 2016-12-30 13:38 UTC (History)
6 users (show)

See Also:
Tags:
Is this bug a regression?: ---
Last known good build:


Attachments
Working windows console app and Xamarin iOS app with premature end of stream (237.47 KB, application/x-zip-compressed)
2016-10-03 20:37 UTC, mfulker
Details

Description mfulker 2016-10-03 20:37:53 UTC
Created attachment 17850 [details]
Working windows console app and Xamarin iOS app with premature end of stream

When decompressing certain compressed files, reading the DeflateStream byte-by-byte with ReadByte() will omit the last few bytes, seemingly hitting end-of-stream prematurely.
On the same file, using DeflateStream.CopyTo(otherStream) copies the data correctly.

(I've also encountered a problem with EndOfStreamException when trying to use a BinaryReader on the DeflateStream for the same files.)

I'm not sure why only certain files cause it or what the pattern is in the failing files.


In the attached zip are solutions for a Xamarin iOS app and a Windows Console App.
Each has two compressed file resources that they attempt to decompress: "compressed.bin" and "compressed2.bin".
The decompressed size is calculated for each, both via CopyTo into a MemoryStream, and by reading to the end with ReadByte().
The only incorrect case is when reading "compressed.bin" with ReadByte on iOS, which yields 125377 bytes instead of 125387 bytes.
Using ReadByte on Windows or using CopyTo will yield the correct deflated size for the first file.
The other file "compressed2.bin" decompresses correctly on both platforms with both approaches.
Comment 1 mfulker 2016-10-04 21:30:56 UTC
I discovered that appending some zero-value bytes to the end of the file will fix every case of the issue I've seen.

Doing that as a workaround shouldn't cause any other output problems with the Deflate algorithm, should it?
(My knowledge of the Deflate specification is that it works on a block-by-block basis and that each block has an "is last block" header bit, so I don't expect superfluous bytes in input to cause extra bytes on output or anything like that.)

As a temporary workaround (and depending on ETA for a real fix), I might create a read-only Stream wrapper that appends a small number of superfluous bytes to the end of the wrapped FileStream (without actually modifying the file).
Comment 2 mfulker 2016-10-05 17:44:54 UTC
I actually discovered one particular (very small) compressed file also has incorrect results even with CopyTo:

data:application/octet-stream;base64,7cWxCQAgDACwpeBjgqsgXiHU0fd9QzBLErX1EQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADepcxcuU/atm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm3btm37zy8=


The correct decompressed length should be 81942 bytes according to windows.
The decompressed length when using CopyTo on iOS is 81921.
The decompressed length when reading byte-by-byte on iOS is 81697.
Comment 3 Rolf Bjarne Kvinge [MSFT] 2016-10-06 06:16:45 UTC
This might be related to bug #34916.
Comment 4 João Matos 2016-10-27 19:37:20 UTC
https://github.com/mono/mono/pull/3844
Comment 6 Marek Safar 2016-10-28 10:48:01 UTC
@joao could you merge it to Mono 4.8 as well

Note You need to log in before you can comment on or make changes to this bug.