Linq-To-Sql Batch Updating with Duplicates

Question

I'm updating a database with LINQ-To-SQL in C# .NET 3.5, but the data I'm using isn't always good. There's a good chance there will be duplicates from the inserted data and the database.

At the moment, I'm using this, where dataSetOne , dataSetTwo and dataSetThree are a list of the models to update:

AppDataContext db = new AppDataContext();

db.DatasetOne.InsertAllOnSubmit(dataSetOne);
db.DatasetTwo.InsertAllOnSubmit(dataSetTwo);
db.DatasetThree.InsertAllOnSubmit(dataSetThree);

try
{
    db.SubmitChanges(ConflictMode.ContinueOnConflict);
}

catch (Exception e)
{
    CsvToImsConverter.Log.Error(e.Message);

    foreach (ObjectChangeConflict occ in db.ChangeConflicts)
    {
        MetaTable metatable = db.Mapping.GetTable(occ.Object.GetType());
        Console.WriteLine("Error in Table: {0}", metatable.TableName);
    }
}

And this works fine. The problem I have is that if there are any duplicates, the program doesn't update, throwing a primary key violation. I want to insert the values that don't have a primary key violation, and update those that don't.

I can search the database for duplicates, but since there can be thousands of records on the database and hundreds I'm attempting to insert, that could be very costly. Is there any way of catching the primary key violation error, updating this record, and continuing? Since there won't be many duplicates, this would be a far better solution. Using LINQ-To-SQL isn't mandatory.

Would I be better off using a foreach loop and insert each object in the list with a catch statement?

Thanks in advance.

Answer 1

You are worried about performance and at the same time using Linq-2-sql for batch inserts/updates. That is not very consistent since linq-2-sql is not suitable for batch operations (at least not if you need them to be fast).

Don't be fooled by InsertAllOnSumbit... Linq-2-sql will create a separate insert statement for each insert. This is by itsself already very slow.

You are much better of bulkinserting your datasetOne/Two/Three in a staging table (eq using SqlBulkCopy) and after that use a merge statement to move to your target table.