Why is DBSet.Find getting slower and slower?

Question

I'm accessing a mysql database with entity framework in a desktop application. The database is installed on the same system and I am the only one using it now.

I've got a method that checks whether objects from a certain collection docs are in the database already and if they are not they would be added to the database and at the end the save method on DBContext would be executed. The method is executed many times in a loop in my program.

I have noticed that the find speed gets slower and slower with every execution of the method, even though the number of objects to find is fairly constant (around 500 each time). What could be the reason?

The code looks more or less like this:

TimeSpan timeToFind = new TimeSpan();

foreach (var docFromResult in docs)
{
    DateTime operationStart = DateTime.Now;
    var existingDocument =
       db.VaStDocuments.Find(docFromResult.Id, docFromResult.OwnerId, docFromResult.Year);
    timeToFind += DateTime.Now - operationStart;
    if (existingDocument != null && docFromResult.Hash.Equals(existingDocument.Hash))
    {
        continue;
    }
    if (existingDocument == null)
    {
        db.VaStDocuments.Add(docFromResult);
    }
    else if (!docFromResult.Hash.Equals(existingDocument.Hash))
    {
        existingDocument.Hash = docFromResult.Hash;
        existingDocument.IsNew = true;
        existingDocument.Text = null;
    }
    docsForNotification.Add(docFromResult);
}
if (docsForNotification.Any())
{
  db.SaveChangesDisplayValidationErrors();
}

So it is a very rare case that the document will not be found in the database so basically the only database activity here is the Find method.
timetoFind generally increases with every execution of the method. starting with maybe 0.5-2s reaching 120s after maybe 20-30 loops.

Answer 1

Your DbContext keeps track of the items you retrieved and the changes you made to these retrieved items.

This is to make it possible to add / update / remove these items without having to communicate with the database for every action. Only if the SaveChanges is called, the changes are communicated with the database.

This enables you to use newly added or changed object before these changes are committed to the database.

Consider a one-to-many relation ship Customer - Orders: a Customer has zero or more Orders, every Order belongs to exactly one Customer.

Now you can first introduce a new Customer. After that you can Introduce the orders, while the Customer has no Id yet:

using (var dbContext = new MyDbContext(...))
{   // Introduce a Customer:
    Customer customerToAdd = GetCustomerData();
    var addedCustomer = dbContext.Customers.Add(customerToAdd);

    // Introduce an order for this Customer
    Order addedOrder = dbContext.Orders.Add(new Order()
    {
        // addedCustomer has no Id yet, we can't use addedCustomer.Id
        Order.Customer = addedCustomer;
        ...
    });
    dbContext.SaveChanges();
}

Because DbContext kept track of the added customer you are able to use it before the changes are saved.

It also enables you to add the order without adding the Customer First:

var notAddedCustomer = new Customer() {...}
var order1= dbContext.Orders.Add(new Order()
{
    // this order belongs to a new customer that has not been added yet:
    Customer = notAddedCustomer
}

dbContext detects that the customer has not been added yet, and will Add it itself. If you'd create another order for the same notAddedCustomer , then dbContext must be able to detect that this customer was added during the introduction of the first order.

Also, if you remove a Customer, and try to give him a new Order, the dbContext needs to detect that this Customer is removed, and that the new Order cannot be added. Similarly, if you remove a Customer after adding some Orders, dbContext should detect this.

This is why dbContext needs to keep track of changes. Alas this slows down usage if you have a lot of changed objects. Most users of dbContext let it only live for a fairly short time with a fairly small amount of changes.

If you need to do a lot of changes before you do your SaveChanges, and you are certain that you don't need to detect any of them before you do the actual SaveChanges, you can switch off your Tracking by using Queryable.AsNoTracking . This dramatically increases the speed of query and adding the items. The disadvantage is that you have to use Ids instead of objects when using related objects.