如何提高GZIP性能

Question

Currently I do have the problem that this piece of code will be called >500k of times. The size of the compressed byte[] is less than 1KB. Every time the method is called all of the streams has to been created. So I am looking for a way to improve this code.

private byte[] unzip(byte[] data) throws IOException, DataFormatException {

    byte[] unzipData = new byte[4096];

    try (ByteArrayInputStream in = new ByteArrayInputStream(data);
         GZIPInputStream gzipIn = new GZIPInputStream(in);
         ByteArrayOutputStream out = new ByteArrayOutputStream()) {

        int read = 0;
        while( (read = gzipIn.read(unzipData)) != -1) {
            out.write(unzipData, 0, read);
        }

        return out.toByteArray();
    }
}

I already tried to replace ByteArrayOutputStream with a ByteBuffer , but at the time of creation I don't know how many bytes I need to allocate.

Also, I tried to use Inflater but I stumbled across the problem descriped here .

Any other idea what I could do to improve the perfomance of this code.

UPDATE#1

• Maybe this lib helps someone.
• Also there is an open JDK-Bug .

Answer 1

1. Profile your application, to be sure that you're really spending optimizable time in this function. It doesn't matter how many times you call this function; if it doesn't account for a significant fraction of overall program execution time, then optimization is wasted.

2. Pre-size the ByteArrayOutputStream . The default buffer size is 32 bytes, and resizes require copying all existing bytes. If you know that your decoded arrays will be around 1k, use new ByteArrayOutputStream(2048) .

3. Rather than reading a byte at a time, read a block at a time, using a pre-allocated byte[] . Beware that you must use the return value from read as an input to write . Better, use something like Jakarta Commons IOUtils.copy() to avoid mistakes.

Answer 2

I'm not sure if it applies in your case, but I've found incredible speed difference when comparing using the default buffer size of GZIPInputStream vs increasing to 65536.

example: using a 500M input file ->

new GZIPInputStream(new FileInputStream(path.toFile())) // takes 4 mins to process

vs

new GZIPInputStream(new FileInputStream(path.toFile()), 65536) // takes 10s

J

More details can be found here http://java-performance.info/java-io-bufferedinputstream-and-java-util-zip-gzipinputstream/

Both BufferedInputStream and GZIPInputStream have internal buffers. Default size for the former one is 8192 bytes and for the latter one is 512 bytes. Generally it worth increasing any of these sizes to at least 65536.

Answer 3

You can use the Inflater class method reset() to reuse the Inflater object without having to recreate it each time. You will have a little bit of added programming to do in order to decode the gzip header and perform the integrity check with the gzip trailer. You would then use Inflater with the nowrap option to decompress the raw deflated data after then gzip header and before the trailer.