Speed up the insert of 70,000 rows (SProc, C#)

Question

I have a list of 70,000 custom objects in C#, that hold four values. I need to insert all of these into a SQL Server table, if they don't already exist in a different table (same DB). I currently have a single SProc that inserts an individual record, with a foreach loop calling the SProc for each item in my list. Inserting 70,000 is taking about 14 minutes, which feels very slow to me, so is there a better approach?

The rough SProc is:

CREATE Procedure [dbo].[up_UpdateOverrideTable](
    @ItemId varchar(100),
    @ItemValue1 decimal(8,5),
    @ItemValue2 decimal(8,5),
    @Source varchar(2)
)
AS
BEGIN
    DECLARE @LocItemId varchar(100)
    SET @LocItemId = @ItemId
    DECLARE @LocItemValue1 decimal(8,5)
    SET @LocItemValue1 = @ItemValue1
    DECLARE @LocItemValue2 decimal(8,5)
    SET @LocItemValue2 = @ItemValue2
    DECLARE @LocSource varchar(2)
    SET @LocSource = @Source

    DELETE FROM OverrideTable WHERE ItemId = @LocItemId

    IF EXISTS (SELECT ItemId FROM InitialTable WHERE ItemId = @LocItemId)
        INSERT INTO OverrideTable VALUES (@LocItemId, @LocItemValue1, @LocItemValue2, @LocSource)

END
GO

The C# that calls this is also below:

using (SqlConnection conn = GetNewConnection()){
    conn.Open();
    using (var tran = conn.BeginTransaction()){
        using (SqlCommand cmd = new SqlCommand("up_UpdateOverrideTable", conn, tran)){
            cmd.CommandType = CommandType.StoredProcedure;

            try{
                foreach (var item in overrides){
                    cmd.Parameters.Clear();
                    // add parameters here 

                    var response = cmd.ExecuteNonQuery();

                    completedInserts.Add(new OverrideItem(item.Id, Convert.ToBoolean(response)));
                }
            }
            catch (Exception ex){
                tran.Rollback();
                throw;
            }

            tran.Commit();
            conn.Close();
        }
    }
}

Is there something I'm missing here, or is this the best I can hope for? I'm wondering whether creating a DataTable in C# would do the trick?

Answer 1

I'm wondering whether creating a DataTable in C# would do the trick?

Yes, using a DataTable in C# is a step towards speeding up the operation. You then need to use it as a data source for SqlBulkCopy as Matt suggested in comments.

DataTable source = null;//your data source as a DataTable
SqlBulkCopy bulkCopy = new SqlBulkCopy("your connection string");
bulkCopy.DestinationTableName = "your target table name";
await bulkCopy.WriteToServerAsync(source);

Answer 2

You have few sql commands which can slow down the procedure response..

1. DELETE FROM OverrideTable WHERE ItemId = @LocItemId
2. IF EXISTS (SELECT ItemId FROM InitialTable WHERE ItemId = @LocItemId) (Make sure you have index on ItemId)

Unless optimization on db level is not done, C# code can't do anything.

And regarding c# code, I think you should create command inside foreach loop.

try{
    foreach (var item in overrides)
    {
        using (SqlCommand cmd = new SqlCommand("up_UpdateOverrideTable", conn, tran))
        {
            cmd.CommandType = CommandType.StoredProcedure;
                // add parameters here 

            var response = cmd.ExecuteNonQuery();

            completedInserts.Add(new OverrideItem(item.Id, Convert.ToBoolean(response)));
        }
    }
}
catch (Exception ex)
{
    tran.Rollback();
    throw;
}

tran.Commit();
conn.Close();