Improve entity framework insert

Question

I have been dealing with this for 2 days and I can't find solution.

using (TaxablePersonContext context = new TaxablePersonContext(this.ConnectionString))
{
    context.Configuration.AutoDetectChangesEnabled = false;
    foreach(TaxablePerson p in persons) // Persons has always size 1000
    {
      // TaxablePerson has some other properties e.g. Name, VatId, Street,...
      p.RecCreatedBy = "application name";
      p.RecCreatedOn = this.SynchronizationStartDateTime;
      p.RecModifiedBy = "application name";
      p.RecModifiedOn = this.SynchronizationStartDateTime;
      p.RecSyncDate = this.SynchronizationStartDateTime;
      p.RecActive = true;
    }
    DateTime start1 = DateTime.Now;
    context.TaxablePersons.AddRange(persons);
    TimeSpan end1 = DateTime.Now.Subtract(start1);

    DateTime start2 = DateTime.Now;
    context.SaveChanges();
    TimeSpan end2 = DateTime.Now.Subtract(start1);
}

I takes nearly 10 seconds to insert 1000 records and 98 seconds to insert 10.000 records in sql server. Can you please advise what to do to improve Entity framework insert performance. I read this post Fastest Way of Inserting in Entity Framework and included the tips mentioned in this post, but still insert is very slow. I need to insert 260.000 records which takes 52 minutes. I'm inserting in batches of 1000, which upper code demonstrates. Data is read from file, when I hit 1000 records I do synhronization with database. What else can I do? Some people mention that when used setting context.Configuration.AutoDetectChangesEnabled = false; performance improved from minutes to nearly some seconds. What am I missing? I'm using entity framework 6.1.3.

Answer 1

Using sql profiler, I discovered that Entity framework sends queries one by one eg

INSERT INTO MyTable (id, name) VALUES (1, 'Bob')
INSERT INTO MyTable (id, name) VALUES (2, 'Peter')
INSERT INTO MyTable (id, name) VALUES (3, 'Joe')

So actually sql server in this case executes 1000 inserts very slow - nearly 10 seconds (although the execute in transaction). I then constructed insert with multiple values - SQL with many values and insertion of 1000 records took 5 seconds (50% better - earlier 10 seconds). Sql server has a limit of sql parameters you can pass, which is 2100, so this the best you can do with this approach.

private void MultiRecordsInsert(TaxablePersonContext context, List<TaxablePerson> personsToAdd)
{
  List<SqlParameter> parameters = new List<SqlParameter>();
  string firstQuery = @"insert into TaxablePerson (c1, c2, c3) values ";
  string query = firstQuery;

  for (int i = 0; i < personsToAdd.Count; i++)
  {
    query += "(@c1" + i.ToString();
    query += ",@c2" + i.ToString();
    query += ",@c3" + i.ToString() + "),";
    parameters.Add(new SqlParameter("@c1" + i.ToString(), personsToAdd[i].c1));
    parameters.Add(new SqlParameter("@c2" + i.ToString(), personsToAdd[i].c2));
    parameters.Add(new SqlParameter("@c3" + i.ToString(), personsToAdd[i].c3));

    // table has 16 columns (I reduced here for simplicity) so: 2100 / 16 = 131, 
    // used 100
    //
    if (i % 100 == 0)
    {
      query = query.Substring(0, query.Length - 1); // remove last comma
      context.Database.ExecuteSqlCommand(query, parameters.ToArray());

      query = firstQuery;
      parameters = new List<SqlParameter>();
    }
  }

  if (parameters.Count > 0) // what is left
  {
    query = query.Substring(0, query.Length - 1);
    context.Database.ExecuteSqlCommand(query, parameters.ToArray());
  }
}