Can I do a very large insert with Linq-to-SQL?

Question

I've got some text data that I'm loading into a SQL Server 2005 database using Linq-to-SQL using this method (psuedo-code):

Create a DataContext

While (new data exists)
{
    Read a record from the text file

    Create a new Record 

    Populate the record

    dataContext.InsertOnSubmit(record);
}

dataContext.SubmitChanges();

The code is a little C# console application. This works fine so far, but I'm about to do an import of the real data (rather than a test subset) and this contains about 2 million rows instead of the 1000 I've tested. Am I going to have to do some clever batching or something similar to avoid the code falling over or performing woefully, or should Linq-to-SQL handle this gracefully?

Answer 1

It looks like this would work however the changes (and thus memory) that are kept by the DataContext are going to grow with each InsertOnSubmit. Maybe it's adviseable to perform a SubmitChanges every 100 records?

I would also take a look at SqlBulkCopy to see if it doesn't fit your usecase better.

Answer 2

IF you need to do bulk inserts, you should check out SqlBulkCopy

Linq-to-SQL is not really suited for doing large-scale bulk inserts.

Answer 3

You would want to call SubmitChanges() every 1000 records or so to flush the changes so far otherwise you'll run out of memory.

If you want performance, you might want to bypass Linq-To-SQL and go for System.Data.SqlClient.SqlBulkCopy instead.

Answer 4

Just for the record I did as marc_s and Peter suggested and chunked the data. It's not especially fast (it took about an hour and a half as Debug configuration, with the debugger attached and quite a lot of console progress output), but it's perfectly adequate for our needs:

Create a DataContext

numRows = 0;
While (new data exists)
{
    Read a record from the text file

    Create a new Record 

    Populate the record

    dataContext.InsertOnSubmit(record)

    // Submit the changes in thousand row batches
    if (numRows % 1000 == 999)
        dataContext.SubmitChanges()

    numRows++
}

dataContext.SubmitChanges()