Bulk insert records into SQL Server database

Question

I have a text file that contains about a million records. What is the best way to insert them into a SQL Server database from C#?

Can I use BULK INSERT ?

Answer 1

Best way is to use the bcp utility or an SSIS workflow . Those tools have refinements like caching and batching your will miss in a naive implementation. Next best option is BULK INSERT statement, as long as the SQL Server engine itself can reach the file. Last option would be SqlBulkCopy class which allows your app do read the file, maybe process it and transform it, then feed the data as an enumerator to the SqlBulkCopy.

Answer 2

I recently worked on same kind of problem , i realized there are couple solutions to it.

I wrote a BatchProgram (for batch program design read this - http://msdn.microsoft.com/en-us/magazine/cc164014.aspx )

You can use SQL Server Utilities either BCP.exe or OSQL.exe or .net framework supplied SQLBulkCopy class.

I ended up using BCP ( i got a CSV file and used a formatting file and load the data) and OSQL ( i used OSQL where i have to supply a file to the the stored proc )

i also went to .NET Process class and used outputdatarecieved event to log all output of BCP.exe into console (Read this - http://msdn.microsoft.com/en-us/library/system.diagnostics.process.outputdatareceived.aspx ) this worked pretty well.

I also tried to SQLBulkCopy class , but it can be slow if you load data first to datatable ( http://msdn.microsoft.com/en-us/library/ex21zs8x.aspx ) , if you use IDataReader ( http://msdn.microsoft.com/en-us/library/434atets.aspx ) it could be fast .

Since i had millions rows i tried using CSVReader ( http://www.codeproject.com/Articles/9258/A-Fast-CSV-Reader ) which is pretty fast .But down the line there was too much problem in data conversion and i did not have much flexibity in SQL server side .

I ended up using BCP and OSQL.

Answer 3

Is using C# a requirement? The fastest way is to use the bcp command line tool: http://msdn.microsoft.com/en-us/library/ms162802.aspx