Fast approach to C# SqlBulkCopy

Question

I'm reading data from huge csv file (5,000,000 rows and hundreds of columns) and using SqlBulkCopy for writing data into sql table. I'm trying to speed up performance and found WriteToServerAsync method in SqlBulkCopy class. It is much faster than WriteToServer method but cannot see any data in sql table.

public static void InsertDataIntoSQLServerUsingSQLBulkCopy_2(DataTable dtable, string sqlTableName, Int32 batch_size, string connString)
{
    try
    {
        using (SqlBulkCopy bulkCopy = new SqlBulkCopy(connString))
        {
            bulkCopy.DestinationTableName = sqlTableName;

            try
            {
                bulkCopy.BulkCopyTimeout = 0;
                bulkCopy.BatchSize = batch_size;
                bulkCopy.WriteToServerAsync(dtable);
            }
            catch (Exception e)
            {
                Console.WriteLine(e.Message.ToString());
                Environment.Exit(0);
            }
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message.ToString());
        Environment.Exit(0);
    }
}

int batchsize = 0; 
while (!reader.EndOfStream) 
{ 
    string[] line = reader.ReadLine().Split('\t'); 
    datatable.Rows.Add(line); 
    batchsize += 1; 
    if (batchsize == flushed_batch_size) 
    { 
        InsertDataIntoSQLServerUsingSQLBulkCopy_2(dt, tabName, flushed_batch_size, connString);
        dt.Rows.Clear();
        batchsize = 0; 
    } 
    rows += 1; 
}
InsertDataIntoSQLServerUsingSQLBulkCopy_2(dt, tabName, flushed_batch_size, connString);
dt.Rows.Clear();

Is there anything I'm missing in my InsertDataIntoSQLServerUsingSQLBulkCopy_2 method?

Answer 1

My answer doesn't address your performance problems. It's to hopefully help you determine why the data isn't showing up.

Patrick mentioned the first problem in the comments. However, make sure you change the signature to include Task instead of void :

public static async Task InsertDataIntoSQLServerUsingSQLBulkCopy_2(
    DataTable dtable, string sqlTableName, Int32 batch_size, string connString)

Secondly, one potential reason you won't see results - or see inconsistent ones - is because you're using Environment.Exit(0) in your catch blocks.

Environment.Exit is different to a return statement in that it will terminate an application, and all associated threads, immediately. This could be a reason why you're not seeing any data.

Lastly, you need to decide how you want to handle the data import. Are you fine with the import process continuing if the insertion of a row throws an exception? The way you've written that nested try..catch gives the impression that you want it to catch an error but continue. However, the call to Environment.Exit effectively makes that try..catch redundant, so it just becomes code duplication at that point.