Remove duplicates from PreparedStatement batch JDBC

Question

I have a batch which I need to use in order to update a table. This batch is getting fill by an algorithm that I can't change. But it is doing something like this:

String updateFilter = "UPDATE payload_roas SET filtered = TRUE WHERE asn = ? AND prefix= ? AND max_length = ?";
PreparedStatement ps=  connection.prepareStatement(updateFilter);

            for(int i = 0; i < roas.size(); i++) {
               roa = roas.get(i);
               ps.setLong(1, roa.getAsn());
               ps.setObject(2, roa.getPrefix(), OTHER);
               ps.setInt(3, roa.getMax_length());
               ps.setBoolean(4, roa.isWhitelist);
               ps.setBoolean(5, roa.isFilter);
               ps.addBatch();
        }

This batch is quite big ~50K entries.So no surprise that it takes a lot of time when I'm doing executeBatch. However this ps contains lots of duplicates which means it is doing multiple redundant updates. Is there any way to perform a distinct operation on this batch in order to remove this duplicates?

Answer 1

You can try to do something like this:

String updateFilter = "UPDATE payload_roas SET filtered = TRUE WHERE asn = ? AND prefix= ? AND max_length = ?";
PreparedStatement ps=  connection.prepareStatement(updateFilter);
HashSet<String> hashKeys = new HashSet<>();
for(int i = 0; i < roas.size(); i++) {
    roa = roas.get(i);
    String key = roa.getAsn() + roa.getPrefix().toString() + roa.getMax_length() + roa.isWhitelist + roa.isFilter;
    if (!hashKeys.contains(key)) {
        hashKeys.add(key);
        ps.setLong(1, roa.getAsn());
        ps.setObject(2, roa.getPrefix(), OTHER);
        ps.setInt(3, roa.getMax_length());
        ps.setBoolean(4, roa.isWhitelist);
        ps.setBoolean(5, roa.isFilter);
        ps.addBatch();
    }
}