Java Inflater 有时会无限循环

Question

In my application, I'm trying to compress/decompress byte array using java's Inflater/Deflater class. Here's part of the code I used at first:

   ByteArrayOutputStream outputStream = new ByteArrayOutputStream(data.length);  
   byte[] buffer = new byte[1024];  
   while (!inflater.finished()) {  
       int count = inflater.inflate(buffer);  
       outputStream.write(buffer, 0, count);  
   }  

Then after I deployed the code it'll randomly (very rare) cause the whole application hang, and when I took a thread dump, I can identify that one thread hanging

    at java.util.zip.Inflater.inflateBytes(Native Method)
    at java.util.zip.Inflater.inflate(Inflater.java:259)
      - locked java.util.zip.ZStreamRef@fc71443
    at java.util.zip.Inflater.inflate(Inflater.java:280)

It doesn't happen very often. Then I googled everywhere and found out it could be some empty byte data passed in the inflater and finished() will never return true.

So I used a workaround, instead of using

while (!inflater.finished()) 

to determine if it's finished, I used

while (inflater.getRemaining() > 0)

But it happened again. Now it makes me wonder what's the real reason that causes the issue. There shouldn't be any empty array passed in the inflater, even if it did, how come getRemaining() method did not break the while loop? Can anybody help pls? It's really bugging me.

Answer 1

Confused by the same problem, I find this page.

This is my workaround for this, it may helps:

ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
byte[] buffer = new byte[1024];
while (!inflater.finished()) {
    int i = inflater.inflate(buffer);
    if (i == 0) {
        break;
    }
    byteArrayOutputStream.write(buffer, 0, i);
}

Answer 2

The javadoc of inflate :

Uncompresses bytes into specified buffer. Returns actual number of bytes uncompressed. A return value of 0 indicates that needsInput() or needsDictionary() should be called in order to determine if more input data or a preset dictionary is required. In the latter case, getAdler() can be used to get the Adler-32 value of the dictionary required.

So @Wildo Luo was certainly right to check for 0 being returned.

byte[] buffer = new byte[1024];  
while (!inflater.finished()) {  
    int count = inflater.inflate(buffer);  
    if (count != 0 ) {
       outputStream.write(buffer, 0, count);  
    } else {
       if (inflater.needsInput()) { // Not everything read
           inflater.setInput(...);
       } else if (inflater.needsDictionary()) { // Dictionary to be loaded
           inflater.setDictionary(...);
       }
    }
}
inflater.end();

I can only imagine that elsewhere the code is not entirely right, maybe on the compression size. Better first check the general code. There is the Inflater(boolean nowrap) requiring an extra byte, the end() call. Exception handling (try-finally). Etcetera.

For unkown data, unknown occurrences: using a try-catch, find compressed data to check whether it is a data based error, and for testing any solution.

Answer 3

Having the same problem...

What I'm sure about:

1. I'm having an infinite loop, assured with logs printed.
2. inflater.inflate returns 0, and the output buffer size is 0.

My loop is like this (Hive ORC code):

while (!(inflater.finished() || inflater.needsDictionary() ||
             inflater.needsInput())) {
      try {
        int count = inflater.inflate(out.array(),
                                     out.arrayOffset() + out.position(),
                                     out.remaining());
        out.position(count + out.position());
      } catch (DataFormatException dfe) {
        throw new IOException("Bad compression data", dfe);
      }
    }

After the out buffer is consumed and its remaining size is 0, the loop will infinitely run.

But I'm not sure about whether it's orc or zlib caused this. On orc side, it fills original data with the same compression buffer size then do the compression, so theoretically it's not possible I get an compressed chunk larger than the buffer size. Possibilities may be zlib or hardware.

That being said, break the loop when count == 0 is dangerous, since there may be still uncompressed data in the inflator.