我如何gzip一个InputStream并返回一个InputStream？

Question

I am starting with a response from a HTTP request:

InputStream responseInputStream = response.getEntityInputStream()

I need to gzip that response so I can upload it to s3 and save it compressed:

this.s3.putObject(new PutObjectRequest(bucketName, key, gzippedResponseInputStream, meta));

I am aware that I can get the byte[] array out of responseInputStream and then gzip them into a new InputStream . However, that can be very inefficient with a large amount of data.

I know that there have been similar questions asked on SO, but I have not found anything that seems to address the specific need of starting with an InputStream and finishing with a gzipped InputStream .

Thanks for any help!

Answer 1

public final class Example {
    public static void main(String[] args) throws IOException, InterruptedException {
        final PipedInputStream inputStream = new PipedInputStream();
        final PipedOutputStream outputStream = new PipedOutputStream(inputStream);

        Thread compressorThread = new Thread() {
            @Override
            public void run() {
                try (FileInputStream dataSource = new FileInputStream(args[0])) {
                    try (GZIPOutputStream sink = new GZIPOutputStream(outputStream)) {
                        final byte[] buffer = new byte[8 * 1024];
                        for (int bytesRead = dataSource.read(buffer); bytesRead >= 0; bytesRead = dataSource.read(buffer)) {
                            sink.write(buffer, 0, bytesRead);
                        }
                    }
                } catch (IOException ex) {
                    //TODO handle exception -> maybe use callable + executor
                }
            }
        };
        compressorThread.start();

        try (FileOutputStream destination = new FileOutputStream(args[1])) {
            final byte[] buffer = new byte[8 * 1024];
            for (int bytesRead = inputStream.read(buffer); bytesRead >= 0; bytesRead = inputStream.read(buffer)) {
                destination.write(buffer, 0, bytesRead);
            }
        }

        compressorThread.join();
    }

}

You are right, my previous example was wrong. You can use piped streams. The catch here is that you cannot use the input and output stream from the same thread. Also don't forget to join() on the writing thread. You can test my example by supplyng two parameters:

• args[0] -> the source file
• args[1] -> the destination to write the compressed content

PS: @11thdimension was a few minutes faster with his piped stream solutions, so if you find this helpful please accept his answer

Answer 2

I think you're looking for a PipedInputStream

Here's how it can be done.

public InputStrema getGZipStream() {
    final PipedOutputStream pos = new PipedOutputStream();
    PipedInputStream pis = new PipedInputStream();

    try (final InputStream responseInputStream = response.getEntityInputStream();
    ){
        pis.connect(pos);

        Thread thread = new Thread() {
            public void run () {
                startWriting(pos, responseInputStream);
            }
        };
        thread.start();
    } catch(Exception e) {
        e.printStackTrace();
    }

    return pis;
}

public void startWriting(OutputStream out, InputStream in) {
    try (GZIPOutputStream gOut = GZIPOutputStream(out);) {
        byte[] buffer = new byte[10240];
        int len = -1;
        while ((len = in.read(buffer)) != -1) {
            gOut.write(buffer, 0, len);
        }
    } catch (Exception ex) {
        ex.printStackTrace();
    } finally {
        try {
            out.close();
        } catch( Exception e) {
            e.printStackTrace();
        }
    }
}

I haven't tested this code, please let me know if this works.