EF Core IQueryable、Count() 效率和最佳实践

Question

I'm writing queries (CQRS pattern) using EF Core 6 that will be consumed by my controller. Besides of the view models I would also like to return some additional data to the client to enable proper pagination (eg. total count, returned count and remaining count).

My original query looked like this:

_posts //DbSet
    .Where(x => x.Status.Equals(PostStatus.Published))
    .OrderByDescending(x => x.CreatedAt)
    .Where(x => x.CreatedAt < createdAtCursor) //cursor pagination based on created at date
    .Include(x => x.Translations
        .Where(x => x.Language.Equals(language)))
    .Where(x => x.Translations != null) //excluding all posts without translation in a given language
    .Take(pageSize)
    .Select(x => x.ToPreviewModel())
    .AsNoTracking();

So I came up with the following solutions:

Option 1: Split query, run CountAsync() in-between

var allPublishedPosts = _posts
        .Where(x => x.Status.Equals(PostStatus.Published))
        .Include(x => x.Translations
            .Where(x => x.Language.Equals(language)))
        .Where(x => x.Translations != null);

int totalCount = await allPublishedPosts
    .CountAsync()
    .ConfigureAwait(false);

var olderThanPublishedPosts = allPublishedPosts
    .Where(x => x.CreatedAt < createdAtCursor);

int olderThanCount = await olderThanPublishedPosts
    .CountAsync()
    .ConfigureAwait(false);

var returnedPublishedPosts = olderThanPublishedPosts
     .OrderByDescending(x => x.CreatedAt)
     .Take(pageSize)
     .Select(x => x.ToPreviewModel())
     .AsNoTracking();

int returnedCount = await returnedPublishedPosts
    .CountAsync()
    .ConfigureAwait(false);

Here I'm worried about Include() being on top of the query - I guess that could be quite expensive to run this command on a larger set than in original query and I also need to call db 3 times and wait for every CountAsync() one by one to finish.

Option 2: Split query, re-write it to take DbContext // DbSet as a parameter and run CountAsync() in parallel on multiple contexts

IQueryable<PostBase> allPublishedPostsQuery(DbSet<PostBase> posts) =>
    posts
        .Where(x => x.Status.Equals(PostStatus.Published))
        .Include(x => x.Translations
            .Where(x => x.Language.Equals(language)))
        .Where(x => x.Translations != null);

//...so on for the other parts of the query

int[]? countResults;
using (var context1 = _contextFactory.CreateDbContext())
using (var context2 = _contextFactory.CreateDbContext())
using (var context3 = _contextFactory.CreateDbContext())
{
    countResults = await Task.WhenAll(
        allPublishedPostsQuery(context1.Posts)
            .CountAsync(),
        olderThanPublishedPostsQuery(context2.Posts)
            .CountAsync(),
        returnedPublishedPostsQuery(context3.Posts)
            .CountAsync())
        .ConfigureAwait(false);
};

int totalPostsCount = countResults[0];
int olderThanPostsCount = countResults[1];
int returnedPostsCount = countResults[2];

Looks like an overkill to me, but I don't know performance-wise. It's a pity we cannot do it on a single context.

Option 3: ??? Ideally I would like to do it in a one complete query, but I'm not sure if this is possible.

Eventually I could leave all the calculations to the front-end and provide only total count, but still I'm curious for the future how to solve it in a best way and possibly improve efficiency of the query itself.

Answer 1

When it comes to pagination, most controls just want a row count along with a page size and page #. For the most part EF is pretty efficient when you tell it to fetch a count. There shouldn't be a need to execute multiple counts. I'll typically just leave off OrderBy* clauses off the query, take the count, then append the OrderBy* clauses and fetch the page of data.

That said, with very large datasets and potentially dynamic criteria used to filter that data, count queries can get fairly expensive. If you are facing a situation where you're potentially looking at very large count values and slow count queries, What I can suggest is a bit of a "cheat":

Take a situation where you have a pagination control set to a page size of 50 and you display 10 page selectors at a time. 1-10 then a >> if more pages.

Build the query without OrderBy/Pagination. Based on the selected page number, determine how many sets of 10 pages need to be loaded:

var pages = ((pageNumber / 10) +1) * 10;

So for instance if loading page 1, pages = 10. If loading page 11 or 12, pages = 20.

Next, limit your total rows based on the pages needed + 1 row, and base your count on this figure.

var count = query.Take(pages * pageSize + 1).Count();

Lastly, check the count against pages * pageSize + 1, if it is equal you have more pages worth to load, otherwise the count reflects the actual number of records.

What this gives you is an adaptive page count. When loading the first page of results it will Count over a maximum of 500 rows (page size of 50, 10 pages). If there are <= 500 rows the pagination control will display the correct # of pages and we can display the actual count. If there are > 500 rows then the pagination will display 10 pages plus expects an 11th page (rendering in the next page >> control), and we display the row count as something like "500+" or "at least 500" rather than the specific count. Where the client may want to get an accurate row count for whatever reason, I will render this as a hyperlink that will actually do a full count and update the total. If the user selects the >> next page, requesting page #11, the count check bumps to limit at 20 pages rather than 10. If there are more than 1000 results the rendering would show pages 11-20 /w the >> next page, and a row count of "1000+".

The limitation of this approach is that you cannot give the user the ability to go to a specific page or show the accurate row count for every query unless they explicitly request it. But this is weighed against improving the most typical scenarios where users expect to search for and find their results on the first page or few pages, and would rarely go beyond the 10 pages before refining their search criteria. The old adage of "the best place to hide a dead body is on page 2 of Google's search results." Row counts and accurate pagination results for very large or complex data sets can be a rather expensive cost for little benefit.