如何通过实体框架优化插入多条记录（使用存在检查）

Question

I have a folder filled with about 200 csv files, each containing about 6000 rows of data containing mutual fund data. I have to copy those comma separated data into the database via Entity Framework.

The two major objects are Mutual_Fund_Scheme_Details and Mutual_Fund_NAV_Details .

• Mutual_Fund_Scheme_Details - this contains columns like Scheme_Name, Scheme_Code, Id, Last_Updated_On.

• Mutual_Fund_NAV_Details - this contains Scheme_Id (foreign key), NAV, NAV_Date.

Each line in the CSV contains all of the above columns so before inserting, I have to -

1. Split each line.
2. Extract first the scheme related data and check if the scheme exists and get id. If it does not exist then insert the scheme details and get id.
3. Using the id obtained from step 2, check if an entry for NAV exists for the same date. If not, then insert it else skip it.
4. If an entry is inserted in Step 3 then the Last_Updated_On date might need to be updated for the scheme with the NAV date (depending on it is newer than existing value)

All the exists checks are done using ANY linq extension method and all the new entries are inserted into the DbContext but the SaveChanges method is called only at the end of processing of each file. I used to call it after each insert but that just takes even longer than right now.

Now since, this involves at least two exists checks, at the most two inserts and one update, the insertion of each file is taking too long close to 5-7 minutes per file. I am looking for suggestions to improve this. Any help would be useful.

Specifically, I am looking to:

1. Reduce the time it takes to process each file
2. Decrease the number of individual exists check (if I can possibly club them in some way)
3. Decrease individual inserts/updates (if I can possibly club them in some way)

Answer 1

It's going to be hard to optimize it with EF. Here is a suggestion:

1. Once you process the whole file (~6000) do the exists check with .Where( x => listOfIdsFromFile.Contains(x.Id)) . This should work for 6000 ids and it will allow you separate inserts from updates.