inserting to SqlLite takes too long

Question

I insert data like this but it takes too long

For 34,000 records it took 20 minutes !! (for example if I insert into SQL Server CE it took only 3 minutes)

conn_sql = new SQLiteConnection(conn_str2);
conn_sql.Open();
cmd_sql = conn_sql.CreateCommand();

for (int i = 0; i < iTotalRows; i++)
{
    try 
    { 
        Makat = dsView.Tables["Items"].Rows[i]["Makat"].ToString().Trim(); 
    }
    catch { Makat = ""; }

    try 
    { 
        Barcode = dsView.Tables["Items"].Rows[i]["Barcode"].ToString().Trim(); 
    }
    catch { Barcode = ""; }

    try 
    { 
        Des = dsView.Tables["Items"].Rows[i]["Des"].ToString().Trim(); 
    }
    catch { Des = ""; }

    try 
    { 
         Price = dsView.Tables["Items"].Rows[i]["Price"].ToString().Trim(); 
    }
    catch { Price = ""; }

    SQL = "INSERT INTO Catalog(Makat,Barcode,Des,Price)VALUES('" + Makat + "','" + Barcode + "','" + Des + "','" + Price + "')";
    cmd_sql.CommandText = SQL;
    cmd_sql.CommandType = CommandType.Text;

    cmd_sql.ExecuteNonQuery();

    //cmd_sql.Dispose();
}

How to insert faster ?

Answer 1

SQLite implicitly wraps queries within a transaction. Beginning and committing transactions in a loop could be slowing things down. I think you should get a significant boost of speed if you start a transaction and commit it once the loop is completed:

conn_sql.Open();
using(var tran = conn_sql.BeginTransaction()) // <--- create a transaction
{
     cmd_sql = conn_sql.CreateCommand();
     cmd_sql.Transaction = tran;   // <--- assign the transaction to the command
              
     for (int i = 0; i < iTotalRows; i++)
     {
          // ...
          cmd_sql.CommandText = SQL;
          cmd_sql.CommandType = CommandType.Text;
          cmd_sql.ExecuteNonQuery();
          //cmd_sql.Dispose();

     }
     tran.Commit(); // <--- commit the transaction
} // <--- transaction will rollback if not committed already

Answer 2

At first, try to use StringBuilder for the string concatenation. Secondly, you are making 34k requests and this is slow, try to make less amount of requests. Let say using StringBuilder concatenate 5k insert statements in one and fire it in transaction, do this until all your data will not be saved.

Answer 3

If you do it in a single transaction it should be faster:

SqlTransaction transaction;
try
{
    conn_sql = new SQLiteConnection(conn_str2);
    conn_sql.Open();
    cmd_sql = conn_sql.CreateCommand();

    transaction = conn_sql.BeginTransaction();

    for (int i = 0; i < iTotalRows; i++)
    {
        // create SQL string
        cmd_sql.CommandText = SQL;
        cmd_sql.CommandType = CommandType.Text;
        cmd_sql.ExecuteNonQuery();
    }

    transaction.Commit();
}
catch
{
    transaction.Rollback();
}