Bulk Operations with SQL CE / LINQ

Question

I'm trying to write a program to convert a large amount of data from a legacy SQL Express system to a newer system based on SQL CE. Here's a quick snapshot of what's going on:

• Most of the tables in the SQL Express install are small (< 10K records)
• One table is --extremely-- large, and is well over 1 million records

For the smaller tables I can use LINQ just fine -- but the large table gives me problems. The standard way of:

foreach(var dataRow in ...) 
{ 
    table.InsertOnSubmit(dataRow); 
}

database.SubmitChanges();

Is painfully slow and takes several hours to complete. I've even tried doing some simple "bulk" operations to try and eliminate one giant insertion at the end of the loop, ie:

foreach(var dataRow in ...)
{
   if(count == BULK_LIMIT)
   {
      count = 0;
      database.SubmitChanges(); 
   }

   count++;       
   table.InsertOnSubmit(dataRow);
}

// Final submit, to catch the last BULK_LIMIT item block
database.SubmitChanges();

I've tried a variety of bulk sizes, from relatively small values like 1K-5K to larger sizes up to 300K.

Ultimately I'm stuck and the process takes roughly the same amount of time (several hours) regardless of the bulk size.

So - does anyone know of a way to crank up the speed? The typical solution would be to use SqlBulkCopy, but that isn't compatible with SQL CE.

A couple of notes:

• Yes I really do want all the records in SQL CE, and yes I've setup the connection to allow the database to max out at 4 GB.
• Yes I really do need every last of the 1M+ records.
• The stuff in each data row is all primitive, and is a mix of strings and timestamps.
• The size of the legacy SQL Express database is ~400 MB.

Thanks in advance - all help is appreciated!

-- Dan

Answer 1

Use a parameterised INSERT statement: Prepare a command, set the parameter values in a loop and reuse the same command for each INSERT.

Remove any indexes and re-apply after you have performed all INSERTs.

Update : Chris Tacke has the fastest solution here using SqlCeResultset: Bulk Insert In SQLCE