Fastest way to import from Excel to MVC3 Application

Question

I'm working on an import from a CSV file to my ASP.NET MVC3/C#/Entity Framework Application.

Currently this is my code, but I'm looking to optimise:

var excel = new ExcelQueryFactory(file);
var data = from c in excel.Worksheet(0)
            select c;
var dataList = data.ToList();

List<FullImportExcel> importList = new List<FullImportExcel>();
foreach (var s in dataList.ToArray())
{
    if ((s[0].ToString().Trim().Length < 6) && (s[1].ToString().Trim().Length < 7))
    {
        FullImportExcel item = new FullImportExcel();
        item.Carrier = s[0].ToString().Trim();
        item.FlightNo = s[1].ToString().Trim();
        item.CodeFlag = s[2].ToString().Trim();

        //etc etc (50 more columns here)

        importList.Add(item);
    }
}

PlannerEntities context = null;
context = new PlannerEntities();
context.Configuration.AutoDetectChangesEnabled = false;
int count = 0;

foreach (var item in importList)
{
    ++count;
    context = AddToFullImportContext(context, item, count, 100, true);
}

private PlannerEntities AddToFullImportContext(PlannerEntities context, FullImportExcel entity, int count, int commitCount, bool recreateContext)
{
      context.Set<FullImportExcel>().Add(entity);

      if (count % commitCount == 0)
      {
          context.SaveChanges();
          if (recreateContext)
          {
              context.Dispose();
              context = new PlannerEntities();
              context.Configuration.AutoDetectChangesEnabled = false;
          }
      }
      return context;
}

This works fine, but isn't as quick as it could be, and the import that I'm going to need to do will be a minimum of 2 million lines every month. Are there any better methods out there for bulk imports? Am I better avoiding EF altogether and using SQLConnection and inserting that way?

Thanks

Answer 1

I do like how you're only committing records every X number of records (100 in your case.)

I've recently written a system that once a month, needed to update the status of upwards of 50,000 records in one go - this is updating each record and inserting an audit record for each updated record.

Originally I wrote this with the entity framework, and it took 5-6 minutes to do this part of the task. SQL Profiler showed me it was doing 100,000 SQL queries - one UPDATE and one INSERT per record (as expected I guess.)

I changed this to a stored procedure which takes a comma-separated list of record IDs, the status and user ID as parameters, which does a mass-update followed by a mass-insert. This now takes 5 seconds.

In your case, for this number of records, I'd recommend creating a BULK IMPORT file and passing that over to SQL to import.

http://msdn.microsoft.com/en-us/library/ms188365.aspx

Answer 2

For large number of inserts in SQL Server Bulk Copy is the fastest way. You can use the SqlBulkCopy class for accessing Bulk Copy from code. You have to create an IDataReader for your List or you can use this IDataReader for inserting generic Lists I have written.

Answer 3

Thanks to Andy for the heads up - this was the code used in SQL, with a little help from the ever helpful, Pinal Dave - http://blog.sqlauthority.com/2008/02/06/sql-server-import-csv-file-into-sql-server-using-bulk-insert-load-comma-delimited-file-into-sql-server/ :)

DECLARE @bulkinsert NVARCHAR(2000)
DECLARE @filepath NVARCHAR(100)
set @filepath = 'C:\Users\Admin\Desktop\FullImport.csv'
SET @bulkinsert = 
    N'BULK INSERT FullImportExcel2s FROM ''' + 
    @filepath + 
    N''' WITH (FIRSTROW = 2, FIELDTERMINATOR = '','', ROWTERMINATOR = ''\n'')'

EXEC sp_executesql @bulkinsert

Still got a bit of work to do to work it into the code, but we're down to 25 seconds for 50000 rows instead of an hour, so a huge improvement!