EF Core Slow Bulk Insert (~80k rows)

Question

I have a Save object which has several collections associated. Total size of the objects is as follows:

The relations between objects can be infered from this mapping, and seem correctly represented in the database. Also querying works just fine.

modelBuilder.Entity<Save>().HasKey(c => c.SaveId).HasAnnotation("DatabaseGenerated",DatabaseGeneratedOption.Identity);
modelBuilder.Entity<Save>().HasMany(c => c.Families).WithOne(x => x.Save).HasForeignKey(x => x.SaveId);
modelBuilder.Entity<Save>().HasMany(c => c.Countries).WithOne(x => x.Save).HasForeignKey(x => x.SaveId);
modelBuilder.Entity<Save>().HasMany(c => c.Provinces).WithOne(x => x.Save).HasForeignKey(x => x.SaveId);
modelBuilder.Entity<Save>().HasMany(c => c.Pops).WithOne(x => x.Save).HasForeignKey(x => x.SaveId);
modelBuilder.Entity<Country>().HasOne(c => c.Save);
modelBuilder.Entity<Country>().HasMany(c => c.Technologies).WithOne(x => x.Country).HasForeignKey(x => new {x.SaveId, x.CountryId});
modelBuilder.Entity<Country>().HasMany(c => c.Players).WithOne(x => x.Country).HasForeignKey(x => new {x.SaveId, x.CountryId});
modelBuilder.Entity<Country>().HasMany(c => c.Families).WithOne(x => x.Country).HasForeignKey(x => new {x.SaveId, x.OwnerId});
modelBuilder.Entity<Country>().HasMany(c => c.Provinces).WithOne(x => x.Owner);
modelBuilder.Entity<Country>().HasKey(c => new { c.SaveId, c.CountryId });
modelBuilder.Entity<Family>().HasKey(c => new { c.SaveId, c.FamilyId });
modelBuilder.Entity<Family>().HasOne(c => c.Save);
modelBuilder.Entity<CountryPlayer>().HasKey(c => new { c.SaveId, c.CountryId, c.PlayerName });
modelBuilder.Entity<CountryPlayer>().HasOne(c => c.Country);
modelBuilder.Entity<CountryPlayer>().Property(c => c.PlayerName).HasMaxLength(100);
modelBuilder.Entity<CountryTechnology>().HasKey(c => new { c.SaveId, c.CountryId, c.Type });
modelBuilder.Entity<CountryTechnology>().HasOne(c => c.Country);
modelBuilder.Entity<Province>().HasKey(c => new { c.SaveId, c.ProvinceId });
modelBuilder.Entity<Province>().HasMany(c => c.Pops).WithOne(x => x.Province);
modelBuilder.Entity<Province>().HasOne(c => c.Save);
modelBuilder.Entity<Population>().HasKey(c => new { c.SaveId, c.PopId });
modelBuilder.Entity<Population>().HasOne(c => c.Province);
modelBuilder.Entity<Population>().HasOne(c => c.Save);

I parse the entire save from a file so I can't add all the collections one by one. After the parsing I have a Save with all its associated collections, adding up to 80k objects, none of which are present in the database.

Then, when I call dbContext.Add(save) it takes around 44 seconds to process, with RAM usage going up from 100mb to around 700mb.

Then, when I call dbContext.SaveChanges() (I tried also the regular BulkSaveChanges() method from EF Extensions with no significant difference) it takes an additional 60s, with RAM usage going up to 1,3Gb.

What is going on here? Why so long and so much memory usage? The actual uploading to the database only takes about the last 5 seconds.

PS: I also tried disabling change detection with no effect.

PS2: Actual usage and full code as requested in comments:

public class HomeController : Controller
{
    private readonly ImperatorContext _db;

    public HomeController(ImperatorContext db)
    {
        _db = db;
    }

    [HttpPost]
    [RequestSizeLimit(200000000)]
    public async Task<IActionResult> UploadSave(List<IFormFile> files)
    {
        [...]
        await using (var stream = new FileStream(filePath, FileMode.Open))
        {
            var save = ParadoxParser.Parse(stream, new SaveParser());
            if (_db.Saves.Any(s => s.SaveKey == save.SaveKey))
            {
                 response = "The save you uploaded already exists in the database.";
            }
            else
            {
                 _db.Saves.Add(save);
            }
            _db.BulkSaveChanges();
        }
        [...]
    }

}

Answer 1

EDIT: 1. Make sure it's not the DB that's the issue.

Execute your own command to see how fast it runs.

2. Keep the active Context Graph small by using a new context for each Unit of Work, also try to Turn off AutoDetechChangesEnabled

3.batch a number of commands together

Here is a good article on Entity Framework and slow bulk INSERTs

Answer 2

Download EFCore.BulkExtensions from nugets

Remove "_db.BulkSaveChanges();" and replace "_db.Saves.Add(save);" with this code

_db.Saves.BulkInsert(save);

Answer 3

I would suggest you take a look at N.EntityFrameworkCore.Extension. It is a bulk extension framework for EFCore 6.0.8+

Install-Package N.EntityFrameworkCore.Extensions

https://www.nuget.org/packages/N.EntityFrameworkCore.Extensions

Once you install the nuget package you can simply use BulkInsert() method directly on the DbContext instance. It supports BulkDelete, BulkInsert, BulkMerge and more.

BulkDelete()

var dbcontext = new MyDbContext();  
var orders = dbcontext.Orders.Where(o => o.TotalPrice < 5.35M);  
dbcontext.BulkDelete(orders);

BulkInsert()

var dbcontext = new MyDbContext();  
var orders = new List<Order>();  
for(int i=0; i<10000; i++)  
{  
   orders.Add(new Order { OrderDate = DateTime.UtcNow, TotalPrice = 2.99 });  
}  
dbcontext.BulkInsert(orders);