Slow Batch/Bulk Insert in SQLite

Question

I trying to import data from a csv file into a sqlite table. My test data is only about 8Mb(50,000 rows) and takes about 15 seconds. However production data is almost 400Mb, and takes forever (at least 30mins+, I gave up waiting).

After much research, I discovered the need to do the inserts in a single transaction (that got me to the 15 second import, great advice! :) ) So that's not the problem. (AFAIK)

I'm also using "ExecuteNonQuery() on a parameterized INSERT statement" as per this Robert Simpson post - and numerous variations.

I was just using TextReader.ReadLine() and String.Split('\\t') , then I read somewhere about ReadLine() being slow due to the number of disk reads, so I looked into reading a bufferedStream, and came across this csv reader . But Still no noticible change in performance.

So, I commented out guts of my insert loop and the read happens near instantly - so I am sure the problem is in my inserting. I've tried numerous of variations of creating the parameterised queries + single transaction, but all with near identical results..

Here's the regular version of my code. Thanks in advance, This is driving me nuts! I'm about to try importing to dataset and inserting that?....

using (TextReader tr = File.OpenText(cFile))
{                       
    using (SQLiteConnection cnn = new SQLiteConnection(connectionString))
    {
        string line;
        string insertCommand = "INSERT INTO ImportTable VALUES (@P0,@P1,@P2,@P3,@P4)";

        cnn.Open();
        SQLiteCommand cmd = new SQLiteCommand("begin", cnn);
        cmd.ExecuteNonQuery();

        cmd.CommandText = insertCommand;

        while ((line = tr.ReadLine()) != null)
        {
            string[] items = line.Split('\t');

            cmd.Parameters.AddWithValue("@P0", items[0]);
            cmd.Parameters.AddWithValue("@P1", items[1]);
            cmd.Parameters.AddWithValue("@P2", items[2]);
            cmd.Parameters.AddWithValue("@P3", items[3]);
            cmd.Parameters.AddWithValue("@P4", items[4]);
            cmd.ExecuteNonQuery();
        }
        cmd.CommandText = "end";
        cmd.ExecuteNonQuery(); 
    }              
}

Update: I just tried using the insert with the parameters (just hard-coded some values), less than 5 seconds... still not as fast as the articles I've seen...

Also, I'm running a Core2 Duo (3Ghz) with 2G Ram, XP.

Answer 1

So I think I've worked out the problem - or at least found a solution.

Since I'd exhausted all my code options (and it didn't look like anybody had an answer/problem with my code), I decided the problem may lie within the database itself...

I had created my database and tables all within SQLite Manager Firefox Plugin.

So I recreated everything from the commandshell, and BOOM! My import dropped down to just a few seconds!

I knew there was a problem with it being unable to handle 64bit integers (but just used TEXT datatypes). Perhaps there is a problem with SQLite Manager using a different SQLite engine to the .Net version? I don't know.

My next step might be to actually create the db + tables from within my application, instead of having them preprepared... But I'm fairly satisfied with the performance now, so that's not a priority.