python zlib produces a -3 error when inflating data through a Decompress() object

Question

I have created a decoder to essentially parse, decompress and extract a single file from a zlib encoded file downloaded through a urllib2 file-like object. The idea is to utilize as little memory and disk space as possible, so I am using a reader / writer pattern with the "decoder" in the middle to uncompress the data coming from urllib2, feed it into a cpio subprocess and finally write the file data to disk:

with closing(builder.open()) as reader:
    with open(component, "w+b") as writer:
         decoder = Decoder()

         while True:
             data = reader.read(10240)
             if len(data) == 0:
                 break

             writer.write(decoder.decode(data))

        final = decoder.flush()
        if final is not None:
            writer.write(final)

        writer.flush()

The decoder is pretty simple too:

class Decoder(object):
    def __init__(self):
        self.__zcat = zlib.decompressobj()
        # cpio initialisation

    def decode(self, data_in):
        return self.__consume(self.__zcat.decompress(data_in))

    def __consume(self, zcat_data_in):
        # cpio operations
        return data_out

    def flush(self):
        return self.__consume(self.__zcat.flush())

I am seeing an error before anything is even passed to the cpio pipe, so I felt omitting it here was sensible for clarity.

The interesting thing, is that to verify the data could in fact be uncompressed by zlib, I wrote the raw data data_in being passed to decode() to stdout:

def decode(self, data_in):
    sys.stdout.write(data_in)
    return self.__consume(self.__zcat.decompress(data_in))

Then ran:

$ bin/myprog.py 2>/dev/null | zcat - | file -
/dev/stdin: cpio archive

As you can see, zcat was quite happy about the data it was given on stdin and the resultant file is a cpio archive. But the zlib decompress method is reporting:

error: Error -3 while decompressing: incorrect header check

Answer 1

\\x1f\\x9d are the first two bytes of the old Unix compress format. zlib can't help you decompress it. gzip can decompress it just to be compatible with the old compress utility.

You can pull the code from pigz for decompressing that format and use it directly.