slow sqlite insert using the jdbc drivers in java

Question

I just inserted 1million records into a simple sqlite table with five columns. It took a whooping 18 hours in java using the jdbc drivers! I did the same thing in python2.5 and it took less than a minute. The speed for select queries seem fine. I think this is an issue with the jdbc drivers.

Is there a faster driver for sqlite3 in java?

Speed of inserting large numbers of rows is important for my schema migration script, and I'd rather not have to use an external script to do the migrations if I don't have to.

EDIT: fixed with connection.setAutoCommit(false); thanks Mark Rushakoff for hinting me to the solution :)

Answer 1

Did you have your queries autocommitted? That could explain why it took so long. Try wrapping them in a begin / end so that it doesn't have to do a full commit for every insert.

This page explains begin/end transaction, while the FAQ touches on inserts/autocommits.

Answer 2

If you want to further optimize, you can look into batching your insert queries together. So you can change 1 million inserts to 1000 inserts of 1000 batches.