Entity Framework bulk insert unreal slow

Question

I am using EF 6. I am trying to insert around 200.000 entities while saving changes to database after each 100 entities.

The issue is it took 11 hours to save 50.000 entities, and it is still running behind. I am running this with WebJobs, and job is published on the same azure webapp as main website. Is the issue because of that and WebJob don't have enough resources, or saving after 100 entities, or the approach?

Method

public void SaveLeadsForBuyer(ISenderModel model)
{
    var rowCounter = 0;

    foreach (var deliveryRecord in model.Customers.Select(customerModel => new DeliveryRecord()
    {
        BuyerId = model.Buyer.Id,
        AspNetUserId = customerModel.Id,
        DeliveryType = model.Buyer.DeliveryType,
        CreatedOn = DateTime.UtcNow
    }))
    {
        ++rowCounter;

        _unit.Repository<DeliveryRecord>().Insert(deliveryRecord);

        _unit.SaveChangesPartially(rowCounter, 100);
    }

    _unit.SaveChanges();
}

Helper

public static class UnitOfWorkHelper
{
    /// <summary>
    /// Helper method triggers SaveChanges() after amount of rows provided through "amount" parameter in method
    /// </summary>
    /// <param name="unit">UnitOfWork object</param>
    /// <param name="count">Current amount of rows</param>
    /// <param name="saveCount">Amount when to save changes to database</param>
    public static void SaveChangesPartially(this IUnitOfWorkAsync unit, int count, int saveCount)
    {
        if (count % saveCount == 0)
        {
            unit.SaveChanges();
        }
    }
}

Answer 1

It's slow because Entity Framework performs a database round trip for every record. So, if you save 200,000 entities then 200,000 database round-trips will be performed which is far to be optimal to save multiple entities.

For this kind of scenario, you need to implement yourself or use a library supporting BulkInsert (which normally execute a SqlBulkCopy under the hood)

There is 3 main library (2 FREE, 1 PRO) which allow Bulk Insert

// Example from Entity Framework Extensions Library
using (var ctx = new EntitiesContext())
{
    ctx.BulkInsert(list);
}

You can read the following article to understand PROS & CONS for every library: Entity Framework - Bulk Insert Library Reviews & Comparisons

Entity Framework Extensions is the library which offers by far the most flexibility (Bulk Insert, Update, Delete, Merge and BulkSaveChanges and supports everything) however it is a PRO version. If you are looking for a free version, I recommend using EntityFramework.BulkInsert, however, it's not supported anymore and doesn't support all associations and inheritances.

Disclaimer : I'm the owner of the project Entity Framework Extensions

EDIT: Answer comment question

I am saving each 100 records, not each record

It doesn't matter if you add one entity or 100 entities to your unit context, Entity Framework saves them one by one (A single insert statement for every record). Just use SQL Profiler with a SQL Server database and you will see what I mean.

EDIT: Answer comment question

great jonathan. is there any way to implement this with ef6 generic uow?

The answer depends on which library you choose to use.

If you use my library, you can create the BulkSaveChanges method or change in your UnitOfWork all "_context.SaveChanges()" by "_context.BulkSaveChanges()"

public void SaveLeadsForBuyer(ISenderModel model)
{
    // ... code ...
    // _unit.SaveChanges();
    _unit.BulkSaveChanges();
}

If you want the best performance and Implement Bulk Insert from my library or a FREE library, I would probably add a method or an extension method (if you cannot change the repository class) named BulkInsert

public class Repository<TEntity> : IRepository<TEntity> where TEntity : class
{
    // ... code ...

    public virtual void BulkInsert(List<TEntity> list)
    {
        _context.BulkInsert(list);
    }
}

Keep in mind BulkInsert directly insert entities without having to call "SaveChanges", it doesn't use the context/change tracker to get the optimal performance.