High performance data insert into SQL Server

Question

I'm inserting about 7 million rows into a table in SQL Server. The table has about 9-10 columns and it has a clustered index on 2 keys (columns) and 3 more unclustered indexes. I'm trying to do high-performance/bulk insert of data, but it gets pretty slow after inserting 3 million records with the disk usage going up to 99%. Here is what I did to speed up this process:

1) Disabled all nonclustered indexes. I didn't disable the 2 column clustered index because data cannot be inserted after the clustered index is disable?

2) I'm using C# SqlBulkCopy to do bulk insert of 5,000 records each time.

3) The table doesn't have any constraints and triggers

Is there anything I should be doing to speed up this data insert process?

Answer 1

Get rid of ALL your indexes. Every time you write with an index on, the physical page has to be re-structured when you write each record. Dump all your indexes using code (DROP INDEX), insert your data and then re-build your indexes using code (CREATE INDEX).

Answer 2

Make sure the database is in recovery mode BULK LOGGED or SIMPLE, at least until you have all the records inserted. This will keep the transaction log file from swelling.

Answer 3

As spender told, your cluster index may be a problem. You may be inserting data such a way that your cluster index is not sorted. So SQL server must be creating lot of pages with only one entry in it and doing optimizing once it sees a lot of sparsely filled packets. Try using auto generated int indexes. If you cannot do that, sort the entire bulk data (based on clustered index column - order is important) before inserting them into SQL. This should minimize the disk usage.