EF Core 在大循环中出现 ABP 性能问题

Question

I have a background job that processes a list of items. However, the larger the list, the time to complete it grows exponentially. Not just that, the further down the list, the app gradually eats up my server's 64gb ram and slows down the server to a crawl.

I use Hangfire as my background job manager, and the job takes a list of id and performs multiple database actions on them. I access the database through ef core, using the repository pattern provided by as.netboilerplate framework.

I assume it is slow because of the growing db context, so here is what I have tried:

• Split the list across multiple jobs.

   var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { await _backgroundJobManager.EnqueueAsync<MyJob, MyJobArgs>(new MyJobArgs(...)) }

• Split the job into multiple unit of work

  var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { using (var uow = UnitOfWorkManager.Begin()) { foreach(var id in split) { //create related entity.... //create barcode.... //link entity to created barcode.... } uow.Complete(); } }

• Split the job into multiple unit of work with new transactions

  var splitIdList = ListExtensions.ChunkBy(idList, 100); foreach (var split in splitIdList) { using (var uow = UnitOfWorkManager.Begin(System.Transactions.TransactionScopeOption.RequiresNew)) { foreach(var id in split) { //create related entity.... //create barcode.... //link entity to created barcode.... } uow.Complete(); } }

None seems to solve the problem, any help would be appreciated, thank you.

EDIT Here is what I do to each entity in the job:

1. I create another entity and link them together.
2. I create a qrcode, which is a running number, and link it to the entity

Operations involved are create, get, insert, and update

Answer 1

I had the same issue. In case this happens to anyone else, here is what I found.

TLDR;

When adding items to a collection of a parent Entity, make sure you release the Entity if those collection items are no longer needed to free memory.

Explination

I have a BackgroundJob processing a file. The File table record holds information regarding the file and a FileExceptions table logs any errors processing any single records. The job opened the File record and passed it into the record processing function.

Record processing function would open a new transaction in the unit of work and call await uow.CompleteAsync(); at the end. When an Exception was added to the FileExceptions, I found that the File record held all of those exceptions in memory.

Here is my original version's pseudo-code

public class FileProcessJob : BackgroundJob<long>, ITransientDependency {
    
    [UnitOfWork]
    public override void Execute(long fileId)
    {
        MyFile file = _repository.FirstOrDefault(fileId);
        
        //Show we are processing the file and update database
        using (var uow = _unitOfWorkManager.Begin(System.Transactions.TransactionScopeOption.RequiresNew))
        {
            file.Status = FileStatus.Processing;
            await _fileRepository.UpdateAsync(bfile);
            await uow.CompleteAsync();
        }


        //Process the file
        ProcessFile(file);


        //Show the file has been processed
        using (var uow = _unitOfWorkManager.Begin())
        {
            file.Status = FileStatus.Processed;
            await _fileRepository.UpdateAsync(file);
            await uow.CompleteAsync();
        }            
    }
    

    private void ProcessFile(MyFile file) {
        foreach(var record in fileRecords) {
            ProcessRecord(file, record);
        }
    }


    private void ProcessRecord(MyFile file, object record) {
        using (var uow = _unitOfWorkManager.Begin(TransactionScopeOption.RequiresNew))
        {
            //...Process the record...

            if(proessingError) _fileExceptionRepository.Insert(exception);
            await _fileManager.UpdateAsync(file);
            await uow.CompleteAsync(); 
        }
    }
}

Because the file object was scoped outside ProcessRecord, the file.Exceptions collection grew. Simple solution was as follows.

Updated

public class FileProcessJob : BackgroundJob<long>, ITransientDependency {
    
    [UnitOfWork]
    public override void Execute(long fileId)
    {
        MyFile file = _repository.FirstOrDefault(fileId);
        
        //Show we are processing the file and update database
        using (var uow = _unitOfWorkManager.Begin(System.Transactions.TransactionScopeOption.RequiresNew))
        {
            file.Status = FileStatus.Processing;
            await _fileRepository.UpdateAsync(bfile);
            await uow.CompleteAsync();
        }


        //Process the file
        ProcessFile(file);


        //Show the file has been processed
        using (var uow = _unitOfWorkManager.Begin())
        {
            /* NOTE: ANY UPDATES DONE TO THIS ENTITY 
            IN ProcessFile OR ProcessRecord WILL HAVE 
            BEEN LOST.  THIS ENTITY IS EXACTLY AS IT WAS
            WHEN IT WAS LAST UPDATED ABOVE */
            file.Status = FileStatus.Processed;
            await _fileRepository.UpdateAsync(file);
            await uow.CompleteAsync();
        }            
    }
    

    private void ProcessFile(MyFile file) {
        foreach(var record in fileRecords) {
            ProcessRecord(file.Id, record);
        }
    }


    private void ProcessRecord(long fileId, object record) {
        using (var uow = _unitOfWorkManager.Begin(TransactionScopeOption.RequiresNew))
        {
            MyFile file = _repository.FirstOrDefault(fileId);

            //...Process the record...

            if(proessingError) _fileExceptionRepository.Insert(exception);
            await _fileManager.UpdateAsync(file);
            await uow.CompleteAsync(); 
        }
    }
}