SQLite DB Insert Very Slow

Question

I am using an SQLite database, and inserting records into it. This takes a hugely long time! I have seen people who say they can process a couple thousand in a minute. I have around 2400 records. Each record takes 30s-2m to complete. Recreating the database is not an option. I have tried to create one transaction different ways. I need to use the timer, because I am using a ProgressBar to show me that something is happening. Here is the code I am using:

string con;
con = string.Format(@"Data Source={0}", documentsFolder);

SQLiteConnection sqlconnection = new SQLiteConnection(con);
SQLiteCommand sqlComm = sqlconnection.CreateCommand();
sqlconnection.Open();
SQLiteTransaction transaction = sqlconnection.BeginTransaction();

Timer timer2 = new Timer();
timer2.Interval = 1000;
timer2.Tick += (source, e) =>
                    {
                        URL u = firefox.URLs[count2];
                        string newtitle = u.title;
                        form.label1.Text = count2 + "/" + pBar.Maximum;
                        string c_urls = "insert or ignore into " + table + " (id,
 url, title, visit_count, typed_count, last_visit_time, hidden) values (" + dbID + ",'" + u.url + "','" 
    + newtitle + "',1,1, " + ToChromeTime(u.visited) + ", 0)";
                        string c_visited = "insert or ignore into " + table2 + " (id,
 url, 
    visit_time, transition) values (" + dbID2 + "," + dbID + "," + 
ToChromeTime(u.visited) + ",805306368)";
                        sqlComm = new SQLiteCommand(c_urls, sqlconnection);
                        sqlComm.ExecuteNonQuery();
                        sqlComm = new SQLiteCommand(c_visited, sqlconnection);
                        sqlComm.ExecuteNonQuery();

                        dbID++;
                        dbID2++;


                        pBar.Value = count2;
                        if (pBar.Maximum == count2)
                        {
                            pBar.Value = 0;
                            timer.Stop();
                            transaction.Commit();
                            sqlComm.Dispose();
                            sqlconnection.Dispose();
                            sqlconnection.Close();
                        }

                        count2++;
                    };
timer2.Start();

What am I doing wrong?

Answer 1

This is what I would address, in order. It may or may not fix the problem, but it won't hurt to see (and it might just do some magic):

1. Ensure the Database is not being contended with updates (from another thread, process, or even timer!). Writers will acquire locks and unclosed/over-long-running transactions can interact in bad ways. (For updates that take "30 seconds to 2 minutes" I would imagine there is an issue obtaining locks. Also ensure the media the DB is on is sufficient, eg local drive.)

2. The transaction is not being used (??). Move the transaction inside the timer callback, attach it to the appropriate SQLCommands, and dispose it before the callback ends. (Use using ).

3. Not all SQLCommand's are being disposed correctly. Dispose each and every one. (The use of using simplifies this. Do not let it bleed past the callback.)

4. Placeholders are not being used. Not only is this simpler and easier to use, but it is also ever so slightly more friendly to SQLite and the adapter.

(Example only; there may be errors in the following code.)

// It's okay to keep long-running SQLite connections.
// In my applications I have a single application-wide connection.
// The more important thing is watching thread-access and transactions.
// In any case, we can keep this here.
SQLiteConnection sqlconnection = new SQLiteConnection(con);
sqlconnection.Open();

// In timer event - remember this is on the /UI/ thread.
// DO NOT ALLOW CROSS-THREAD ACCESS TO THE SAME SQLite CONNECTION.
// (You have been warned.)
URL u = firefox.URLs[count2];
string newtitle = u.title;
form.label1.Text = count2 + "/" + pBar.Maximum;

try {
   // This transaction is ONLY kept about for this timer callback.
   // Great care must be taken with long-running transactions in SQLite.
   // SQLite does not have good support for (long running) concurrent-writers
   // because it must obtain exclusive file locks.
   // There is no Table/Row locks!
   sqlconnection.BeginTransaction();
   // using ensures cmd will be Disposed as appropriate.
   using (var cmd = sqlconnection.CreateCommand()) {
     // Using placeholders is cleaner. It shouldn't be an issue to
     // re-create the SQLCommand because it can be cached in the adapter/driver
     // (although I could be wrong on this, anyway, it's not "this issue" here).
     cmd.CommandText = "insert or ignore into " + table
       + " (id, url, title, visit_count, typed_count, last_visit_time, hidden)"
       + " values (@dbID, @url, 'etc, add other parameters')";
     // Add each parameter; easy-peasy
     cmd.Parameters.Add("@dbID", dbID);
     cmd.Parameter.Add("@url", u.url);
     // .. add other parameters
     cmd.ExecuteNonQuery();
   }
   // Do same for other command (runs in the same TX)
   // Then commit TX
   sqlconnection.Commit();
} catch (Exception ex) {
   // Or fail TX and propagate exception ..
   sqlconnection.Rollback();
   throw;
}

if (pBar.Maximum == count2)
{
    pBar.Value = 0;
    timer.Stop();
    // All the other SQLite resources are already
    // cleaned up!
    sqlconnection.Dispose();
    sqlconnection.Close();
}

Answer 2

I'm not sure if this is your problem, but your general pattern of using ADO.NET is wrong - you shouldn't create new command(s) per each insert (and repeatedly pay for query preparation).

Instead, do the following:

• Before the loop:
  • Create command(s) once.
  • Create appropriate bound parameters.
• In the loop:
  • Just assign appropriate values to the bound parameters.
  • And execute the command(s).

You could also consider using less fine-grained transactions: try putting several inserts in the same transaction to minimize paying for transaction durability .

You might also want to take a look at this post .

Answer 3

You can try one of the following to improve performance :

• Wrap all the inserts in a transaction - Can help in reducing the actual writes to the DB.
• Use WAL - The Write-Ahead-Log is a journaling mode that speeds up writes and enables concurrency. (Not recommended if your DB is in a Network location).
• Synchronous NORMAL - The Synchronous Mode dictates the the frequency at which data is actually flushed to the physical memory (fsync() calls). This can be time taking on some machines and hence the frequency at which this flush occurs is critical. Make sure to explicitly open connections with "Synchronous=NORMAL" ideal for most scenarios. There is a huge difference between Synchronous MODE as FULL and NORMAL (NORMAL is ~1000 times better).

Find more details in a similar post => What changed between System.Data.SQLite version 1.0.74 and the most recent 1.0.113?