Java Heap OutOfMemory on ByteArrayOutputStream for deflater zip bytes

Question

I have a program who read information from database. Sometimes, the message is bigger that expected. So, before send that message to my broker I zip it with this code:

public static byte[] zipBytes(byte[] input) throws IOException {
        ByteArrayOutputStream bos = new ByteArrayOutputStream(input.length);
        OutputStream ej = new DeflaterOutputStream(bos);
        ej.write(input);
        ej.close();
        bos.close();
        return bos.toByteArray();
    }

Recently, i retrieved a 80M message from DB and when execute my code above just OutOfMemory throw on the "ByteArrayOutputStream" line. My java program just have 512 of memory to do all process and cant give it more memory.

How can I solve this?

This is no a duplicate question. I cant increase heap size memory.

EDIT: This is flow of my java code:

rs = stmt.executeQuery("SELECT data FROM tbl_alotofinfo"); //rs is a valid resulset
while(rs.next()){
   byte[] bt = rs.getBytes("data");
   if(bt.length > 60) { //only rows with content > 60. I need to know the size of message, so if I use rs.getBinaryStream I cannot know the size, can I?
      if(bt.length >= 10000000){
         //here I need to zip the bytearray before send it, so
         bt = zipBytes(bt); //this method is above
         // then publish bt to my broker
      } else {
         //here publish byte array to my broker
      }
   }
}

EDIT Ive tried with PipedInputStream and the memory that process consume is same as zipBytes(byte[] input) method.

private InputStream zipInputStream(InputStream in) throws IOException {
        PipedInputStream zipped = new PipedInputStream();
        PipedOutputStream pipe = new PipedOutputStream(zipped);
        new Thread(
                () -> {
                    try(OutputStream zipper = new DeflaterOutputStream(pipe)){
                        IOUtils.copy(in, zipper);
                        zipper.flush();
                    } catch (IOException e) {
                        IOUtils.closeQuietly(zipped);  // close it on error case only
                        
                        e.printStackTrace();
                    } finally {
                        IOUtils.closeQuietly(in);
                        IOUtils.closeQuietly(zipped);
                        IOUtils.closeQuietly(pipe);
                    }
                }
        ).start();
        return zipped;
    }

How can I compress by Deflate my InputStream?

Answer 1

rs.getBytes("data") reads the entire 80 megabytes into memory at once. In general, if you are reading a potentially large amount of data, you don't want to try to keep it all in memory.

The solution is to use getBinaryStream instead.

Since you need to know whether the total size is larger than 10,000,000 bytes, you have two choices:

• Use a BufferedInputStream with a buffer of at least that size, which will allow you to use mark and reset in order to “rewind” the InputStream.
• Read the data size as part of your query. You may be able to do this by using a Blob or using a function like LENGTH .

The first approach will use up 10 megabytes of program memory for the buffer, but that's better than hundreds of megabytes:

while (rs.next()) {
    try (BufferedInputStream source = new BufferedInputStream(
        rs.getBinaryStream("data"), 10_000_001)) {

        source.mark(10_000_000);

        boolean sendToBroker = true;
        boolean needCompression = true;
        for (int i = 0; i <= 10_000_000; i++) {
            if (source.read() < 0) {
                sendToBroker = (i > 60);
                needCompression = (i >= 10_000_000);
                break;
            }
        }

        if (sendToBroker) {
            // Rewind stream
            source.reset();

            if (needCompression) {
                try (OutputStream brokerStream =
                    new DeflaterOutputStream(broker.getOutputStream())) {

                    source.transferTo(brokerStream);
                }
            } else {
                try (OutputStream brokerStream =
                    broker.getOutputStream())) {

                    source.transferTo(brokerStream);
                }
            }
        }
    }
}

Notice that no byte arrays and no ByteArrayOutputStreams are used. The actual data is not kept in memory, except for the 10 megabyte buffer.

The second approach is shorter, but I'm not sure how portable it is across databases:

while (rs.next()) {
    Blob data = rs.getBlob("data");
    long length = data.length();
    if (length > 60) {
        try (InputStream source = data.getBinaryStream()) {
            if (length >= 10000000) {
                try (OutputStream brokerStream =
                    new DeflaterOutputStream(broker.getOutputStream())) {

                    source.transferTo(brokerStream);
                }
            } else {
                try (OutputStream brokerStream =
                    broker.getOutputStream())) {

                    source.transferTo(brokerStream);
                }
            }
        }
    }
}

Both approaches assume there is some API available for your “broker” which allows the data to be written to an OutputStream. I've assumed, for the sake of example, that it's broker.getOutputStream() .