How to increase the performance for inserting data to an table in SSIS

Question

I have a text file containing 3 Lakh (300,000) records with 7 columns. I insert data into a staging table, then perform business logic to insert the data into multiple tables from there.

The file looks like this:

01|000001111|27/04/2011|12/01/2012|ISDF|AB|1      
02|000002222|09/01/2010|29/01/2010|CfGH|CV|1      
03|000003333|19/07/2005|09/07/2007|TBRF|CC|1 

The staging table is called Stagetable .

I am using flat file source and OLEDB destination its taking more than 7 hours to insert data just into the staging table. I need to improve the performance.

I am using SSIS package for the first time.

Any suggestions on how I could improve the performance would be great.

Thanks Prince

Answer 1

The following should help;

-Move your file close to your target (ie on the same box) before processing to avoid.network latency

-Deploy and run your package on from within SQL Server, not Visual Studio

-Use SQL Server destination, not the OLE DB as destination for quicker loading

-Switch off constraint checking on destination component

-Check that the target table does not have expensive triggers occurring on load

-Consider using the bulk insert task if you do not need to perform any transformation

Answer 2

3 lacs is only 300000 and the fields/row does not look particularly wide either. This is pretty small number to cause any problem.

Try to find out where is the bottleneck - at the source or the destination? If source is the bottle neck, you can split the file and stage them into separate but identical staging tables such as stg_1, stg_2 etc., and then load them in parallel.

If destination is the bottleneck, you can use Balanced data distributor, and put them into different staging tables.

Again, all this seems to be an overkill for just 300K rows.

Make sure you do not have any indexes on the staging table(s). We would love to hear from you what all you tried, what worked, and how you resolved.