JDBC - PostgreSQL - batch insert + unique index

Question

I have a table with unique constraint on some field. I need to insert a large number of records in this table. To make it faster I'm using batch update with JDBC (driver version is 8.3-603). Is there a way to do the following:

• every batch execute I need to write into the table all the records from the batch that don't violate the unique index;

• every batch execute I need to receive the records from the batch that were not inserted into DB, so I could save "wrong" records

?

Answer 1

The most efficient way of doing this would be something like this:

• create a staging table with the same structure as the target table but without the unique constraint
• batch insert all rows into that staging table. The most efficient way is to use copy or use the CopyManager (although I don't know if that is already supported in your ancient driver version.

Once that is done you copy the valid rows into the target table:

insert into target_table(id, col_1, col_2)
select id, col_1, col_2
from staging_table 
where not exists (select *
                  from target_table 
                  where target_table.id = staging_table.id);

Note that the above is not concurrency safe! If other processes do the same thing you might still get unique key violations. To prevent that you need to lock the target table.

If you want to remove the copied rows, you could do that using a writeable CTE:

with inserted as (
    insert into target_table(id, col_1, col_2)
    select id, col_1, col_2
    from staging_table 
    where not exists (select *
                      from target_table 
                      where target_table.id = staging_table.id)
    returning staging_table.id;
) 
delete from staging_table
where id in (select id from inserted);

A (non-unique) index on the staging_table.id should help for the performance.