缓存/编译复杂的Linq查询（实体框架）

Question

I have a complex Entity Framework query. My performance bottleneck is not actually querying the database, but translating the IQueryable into query text.

My code is something like this:

var query = context.Hands.Where(...)
if(x)
    query = query.where(...)
....
var result = query.OrderBy(...)
var page = result.skip(500 * pageNumber).Take(500).ToList(); //loong time here, even before calling the DB

do
{
    foreach(var h in page) { ... }

    pageNumber += 1;
    page = result.skip(500 * pageNumber).Take(500).ToList(); //same here
}
while(y)

What can I do? I am using DbContext (with SQLite), so I can't use precompiled query (and even then, it would be cumbersome with query building algorithm like this).

What I basically need, is to cache a "page" query and only change the "skip" and "take" parameters, without recompiling it from the ground up each time.

Answer 1

Your premise is incorrect. Because you have a ToList call at the end of your query you are querying the database where you've indicated, to construct the list. You're not deferring execution any longer. That's why it takes so long. You aren't spending a long time constructing the query, it's taking a long time to go to the database and actually execute it.

If it helps you can use the following method to do the pagination for you. It will defer fetching each page until you ask for the next one:

public static IEnumerable<IEnumerable<T>> Paginate<T>(
    this IQueryable<T> query, int pagesize)
{
    int pageNumber = 0;

    var page = query.Take(pagesize).ToList();
    while (page.Any())
    {
        yield return page;
        pageNumber++;
        page = query.Skip(pageNumber * pagesize)
            .Take(pagesize)
            .ToList();
    }
}

So if you had this code:

var result = query.OrderBy(...);
var pages = result.Paginate();//still haven't hit the database

//each iteration of this loop will go query the DB once to get that page
foreach(var page in pages)
{
    //use page
}

If you want to get an IEnumerable<IQueryable<T>> in which you have all of the pages as queries (meaning you could add further filters to them before sending them to the database) then the major problem you have is that you don't know how many pages there will be. You need to actually execute a given query to know if it's the last page or not. You either need to fetch each page as you go, as this code does, or you need to query the count of the un-paged query at the start (which means one more DB query than you would otherwise need). Doing that would look like:

public static IEnumerable<IQueryable<T>> Paginate<T>(
    this IQueryable<T> query, int pagesize)
{
    //note that this is hitting the DB
    int numPages = (int)Math.Ceiling(query.Count() / (double)pagesize);

    for (int i = 0; i < numPages; i++)
    {
        var page = query.Skip(i * pagesize)
                .Take(pagesize);
        yield return page;
    }
}