How to efficiently stream millions of records from .NET to SQL Server

Question

I have a C# .NET program that processes millions of records a few at a time and then stores them either in a flat file or a SQL Server database.

Currently, to store them in the SQL Server database the program writes all the records to a flat file and then uses bcp to import into SQL Server.

Is there a more efficient method that bypasses the need to write every record to the hard disk before importing into SQL?

Answer 1

Consider using the SqlBulkCopy class in the System.Data.SqlClient namespace. You can use it like this:

using (var conn = new SqlConnection(connectionString))
using (var bulkCopy = new SqlBulkCopy(conn))
{
    conn.Open();
    {
        bulkCopy.DestinationTableName = "dbo.MyTable";
        DataTable dt = BuildMyData();
        try
        {
            bulkCopy.WriteToServer(dt);
        }
        catch (Exception ex){Console.WriteLine(ex.Message);}
    }
}

The WriteToServer method is overloaded and will accept 3 different param data types:

• DataRow[]
• DataTable
• IDataReader