How to test if a PreparedStatement is going to run out of memory

Question

so I have some java code using JDBC to insert data into a PostgreSQL database using a prepared statement which looks a little like this:

PreparedStatement statement = // prep statement

for (int value: values) {

    statement.setInt(1, value);
    statement.addBatch();
}

statement.executeBatch();

The problem is, I'm occasionally hitting the exception java.lang.OutOfMemoryError: Java heap space . I've taken a look around, but I can't find anything on it; how do I test if the statement is about to run out of memory so that I can execute a batch. Something like this:

PreparedStatement statement = // prep statement

for (int value: values) {

    statement.setInt(1, value);
    statement.addBatch();

    if (statement.currentSize() + sizeof(int) > statement.mazSize()) {

        statement.executeBatch();
    }
}

statement.executeBatch();

Thanks for any help.

Answer 1

There is no reliable way to find out in advance whether an instantiation will fail.

However, more to the point: you are probably misusing the JDBC batching facility, which is there to solve the issue of network roundtrip overhead when inserting many rows. Batch sizes of anything above 100 show diminishing returns and may actually result in slowdowns.

So, change your batching policy to use fixed batch size of a smallish, double-digit integer.

Answer 2

I faced same issue with a code similar to yours because it was assumed that input array / list ( values ) size would at max be in triple digits - that was an assumption at code level that developer who coded it had.

And one fine day, array / list came to be close to 14k and this code caused OOM though one thing that I noticed that data successfully got committed to DB ( Its Db2 + Oracle Weblogic Set up ) & few seconds later after few OOMs , system went down.

To fix the issue, I simply put statement.executeBatch(); at a pre decided fixed size like below assuming fixed batch size to be 100,

int counter = 0;
for (int value: values) {
    statement.setInt(1, value);
    statement.addBatch();
    ++counter;
    if(counter >= 100 ){
     statement.executeBatch();
     counter =0;
   }
}
statement.executeBatch();

I am not sure if this has anything to do with drivers too but I would assume this fixed number of batches is going to vary from set up to set up infra wise ( I have a multi node powerful DB2 cluster & Web App is also located in same data center ) and I have tested multiple sizes and 1000 goes fairly fast with out any issues but I reduced the size because this code gets kicked from an online UI app so there could be multiple triggers for same code.

I guess it would depend on both - number of Statement objects that one can keep in memory & how long DB takes to process one submitted batch because code will be waiting that long.