Insert Large CSV file into SQL table using C# for import and manipulation

Question

Looking for some help regarding a C#/SQL project that I'm working on.

I have a csv file with ~14 millions intervals. Here is a header and a sample row

File 1 - F1

SENSORID,EXTSENSORID,READTS,VAL

964543,"987654",20220101000000,.03

There is another csv file which contains the location details of 987654

File 2 - F2

EXTSENSORID, LOCSERV, STARTDATE, FINALDATE, MULT1, MULT2

987654, 000053448E000, 2021-12-01, 2200-01-01, 200, 200

The final csv file includes the customer data related to the location

File 3 - F3

LOCSERV,STARTDATE,FINALDATE,CID,RATE

000053448E000, 2021-12-01, 2200-01-01, 123456, ER

These files need to be used to fill a sql table with the following headers:

Headers: EXTSENSORID, UTCDateTime, Value, LocalDateTime, LOCSERV, CID, Rate, MULT1, MULT2

EXTSENSORID - F1[EXTENSORID]

UTCDateTime - F1[READTS] - This needs some manipulation to convert to DateTime

Value - F1[VAL] * F2[MULT1] - F1[EXTSENSORID] relates to F2[EXTSENSORID]

LocalDateTime - F1[READTS].ToLocalTime

LOCSERV - F2[LOCSERV] - F1[EXTSENSORID] relates to F2[EXTSENSORID]

CID- F3[CID] - F2[EXTSENSORID] relates to F3[EXTSENSORID]

RATE- F3[RATE] - F2[EXTSENSORID] relates to F3[EXTSENSORID]

MULT1- F2[MULT1] - F1[EXTSENSORID] relates to F2[EXTSENSORID]

MULT2- F2[MULT2] - F1[EXTSENSORID] relates to F2[EXTSENSORID]

Right now, I'm using StreamReader to bring each row from F1 into an array. I created a list for F2 & F3. Using linq, I look up the necessary parameters from list_F2 & list_F3. Utilizing the array and parameters from list_F2 & list_F3, I create a NewRow() then add the NewRow() to a datatable. I then do a bulkinsert of the datatable into the SQL table.

The bulk insert takes 2 minutes. However, it takes hours to fill the datatable this way.

Any ideas would be helpful. Thanks.

Answer 1

You need the SqlBulkCopy class.

I have something similar for dealing with large (and often boobytrapped by clients) CSV files. A sublcass of StreamReader parses records and fields. I scan the file once to guess colum types and write a table definition, then create the table, and set up the SqlBulkCopy . A second scan of the file then parses the field values to the identified data types and sends batches to the SqlBulkCopy . Use a thread to be sending one batch to the database while you're adding rows to the next SqlBulkCopy .