Entity Framework: Count() very slow on large DbSet and complex WHERE clause

Question

I need to perform a count operation on this Entity Framework (EF6) data set using a relatively complex expression as a WHERE clause and expecting it to return about 100k records.

The count operation is obviously where records are materialized and therefore the slowest of operations to take place. The count operation is taking about 10 seconds in our production environment which is unacceptable.

Note that the operation is performed on the DbSet directly (db being the Context class), so no lazy loading should be taking place.

How can I further optimize this query in order to speed up the process?

The main use case is displaying an index page with multiple filter criteria but the the function is also used for writing generic queries to the ParcelOrders table as required for other operations in the service classes which might be a bad idea resulting in very complex queries resulting from laziness and might potentially be a future problem. The count is later used for pagination, and a much smaller number of records (eg 500) is actually displayed. This is a database-first project using SQL Server.

ParcelOrderSearchModel is a C#-class that serves to encapsualte query parameters and is used exclusively by service classes in order to call the GetMatchingOrders function. Note that on the majority of calls, the majority of the parameters of ParcelOrderSearchModel will be null.

public List<ParcelOrderDto> GetMatchingOrders(ParcelOrderSearchModel searchModel)
{
        // cryptic id known --> allow public access without login
        if (String.IsNullOrEmpty(searchModel.KeyApplicationUserId) && searchModel.ExactKey_CrypticID == null)
            throw new UnableToCheckPrivilegesException();

        Func<ParcelOrder, bool> userPrivilegeValidation = (x => false);

        if (searchModel.ExactKey_CrypticID != null)
        {
            userPrivilegeValidation = (x => true);
        }
        else if (searchModel.KeyApplicationUserId != null)
            userPrivilegeValidation = privilegeService.UserPrivilegeValdationExpression(searchModel.KeyApplicationUserId);

        var criteriaMatchValidation = CriteriaMatchValidationExpression(searchModel);
    
        var parcelOrdersWithNoteHistoryPoints = db.HistoryPoint.Where(hp => hp.Type == (int)HistoryPointType.Note)
            .Select(hp => hp.ParcelOrderID)
            .Distinct();

        Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
        searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);
       
        // todo: use this count for pagination
}


public Func<ParcelOrder, bool> CriteriaMatchValidationExpression(ParcelOrderSearchModel searchModel)
{
        Func<ParcelOrder, bool> expression =
            po => po.ID == 1;

        expression =
           po =>
           (searchModel.KeyUploadID == null || po.UploadID == searchModel.KeyUploadID)
       && (searchModel.KeyCustomerID == null || po.CustomerID == searchModel.KeyCustomerID)
       && (searchModel.KeyContainingVendorProvidedId == null || (po.VendorProvidedID != null && searchModel.KeyContainingVendorProvidedId.Contains(po.VendorProvidedID)))
       && (searchModel.ExactKeyReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber) == searchModel.ExactKeyReferenceNumber)
       && (searchModel.ExactKey_CrypticID == null || po.CrypticID == searchModel.ExactKey_CrypticID)
       && (searchModel.ContainsKey_ReferenceNumber == null || (po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.ContainsKey_ReferenceNumber))
       && (searchModel.OrKey_Referencenumber_ConsignmentID == null ||
               ((po.CustomerID + "-" + po.ReferenceNumber).Contains(searchModel.OrKey_Referencenumber_ConsignmentID)
               || (po.VendorProvidedID != null && po.VendorProvidedID.Contains(searchModel.OrKey_Referencenumber_ConsignmentID))))
       && (searchModel.KeyClientName == null || po.Parcel.Name.ToUpper().Contains(searchModel.KeyClientName.ToUpper()))
       && (searchModel.KeyCountries == null || searchModel.KeyCountries.Contains(po.Parcel.City.Country))
       && (searchModel.KeyOrderStates == null || searchModel.KeyOrderStates.Contains(po.State.Value))
       && (searchModel.KeyFromDateRegisteredToOTS == null || po.DateRegisteredToOTS > searchModel.KeyFromDateRegisteredToOTS)
       && (searchModel.KeyToDateRegisteredToOTS == null || po.DateRegisteredToOTS < searchModel.KeyToDateRegisteredToOTS)
       && (searchModel.KeyFromDateDeliveredToVendor == null || po.DateRegisteredToVendor > searchModel.KeyFromDateDeliveredToVendor)
       && (searchModel.KeyToDateDeliveredToVendor == null || po.DateRegisteredToVendor < searchModel.KeyToDateDeliveredToVendor);
        return expression;
}

public Func<ParcelOrder, bool> UserPrivilegeValdationExpression(string userId)
{
        var roles = GetRolesForUser(userId);

        Func<ParcelOrder, bool> expression =
            po => po.ID == 1;
        if (roles != null)
        {
            if (roles.Contains("ParcelAdministrator"))
                expression =
                    po => true;

            else if (roles.Contains("RegionalAdministrator"))
            {
                var user = db.AspNetUsers.First(u => u.Id == userId);
                if (user.RegionalAdministrator != null)
                {
                    expression =
                        po => po.HubID == user.RegionalAdministrator.HubID;
                }
            }
            else if (roles.Contains("Customer"))
            {
                var customerID = db.AspNetUsers.First(u => u.Id == userId).CustomerID;
                expression =
                    po => po.CustomerID == customerID;
            }
            else
            {
                expression =
                    po => false;
            }
        }

        return expression;
}

Answer 1

If you can possibly avoid it, don't count for pagination. Just return the first page. It's always expensive to count and adds little to the user experience.

And in any case you're building the dynamic search wrong.

You're calling IEnumerable.Count(Func<ParcelOrder,bool>) , which will force client-side evaluation where you should be calling IQueryable.Count(Expression<Func<ParcelOrder,bool>>) . Here:

    Func<ParcelOrder, bool> completeExpression = order => userPrivilegeValidation(order) && criteriaMatchValidation(order);
    searchModel.PaginationTotalCount = db.ParcelOrder.Count(completeExpression);

But there's a simpler, better pattern for this in EF: just conditionally add criteria to your IQueryable.

eg put a method on your DbContext like this:

public IQueryable<ParcelOrder> SearchParcels(ParcelOrderSearchModel searchModel)
{
        var q = this.ParcelOrders();
        if (searchModel.KeyUploadID != null)
        {
          q = q.Where( po => po.UploadID == searchModel.KeyUploadID );
        }
        if (searchModel.KeyCustomerID != null)
        {
          q = q.Where( po.CustomerID == searchModel.KeyCustomerID );
        }
        //. . .
        return q;
}