Linq query timing out, how to streamline query

Question

Our front end UI has a filtering system that, in the back end, operates over millions of rows. It uses a an IQueryable that is built up over the course of the logic, then executed all at once. Each individual UI component is ANDed together (for example, Dropdown1 and Dropdown2 will only return rows that have both of what is selected in common). This is not a problem. However, Dropdown3 has has two types of data in it, and the checked items need to be ORd together, then ANDed with the rest of the query.

Due to the large amount of rows it is operating over, it keeps timing out. Since there are some additional joins that need to happen, it is somewhat tricky. Here is my code, with the table names replaced:

//The end list has driver ids in it--but the data comes from two different places. Build a list of all the driver ids.
driverIds = db.CarDriversManyToManyTable.Where(
                        cd =>
                            filter.CarIds.Contains(cd.CarId) && //get driver IDs for each car ID listed in filter object
                            ).Select(cd => cd.DriverId).Distinct().ToList();

driverIds = driverIds.Concat(
                    db.DriverShopManyToManyTable.Where(ds => filter.ShopIds.Contains(ds.ShopId)) //Get driver IDs for each Shop listed in filter object
                        .Select(ds => ds.DriverId)
                        .Distinct()).Distinct().ToList();
//Now we have a list solely of driver IDs

//The query operates over the Driver table. The query is built up like this for each item in the UI. Changing from Linq is not an option.
query = query.Where(d => driverIds.Contains(d.Id));

How can I streamline this query so that I don't have to retrieve thousands and thousands of IDs into memory, then feed them back into SQL?

Answer 1

There are several ways to produce a single SQL query. All they require to keep the parts of the query of type IQueryable<T> , ie do not use ToList , ToArray , AsEnumerable etc. methods that force them to be executed and evaluated in memory.

One way is to create Union query containing the filtered Ids (which will be unique by definition) and use join operator to apply it on the main query:

var driverIdFilter1 = db.CarDriversManyToManyTable
    .Where(cd => filter.CarIds.Contains(cd.CarId))
    .Select(cd => cd.DriverId);
var driverIdFilter2 = db.DriverShopManyToManyTable
    .Where(ds => filter.ShopIds.Contains(ds.ShopId))
    .Select(ds => ds.DriverId);
var driverIdFilter = driverIdFilter1.Union(driverIdFilter2);
query = query.Join(driverIdFilter, d => d.Id, id => id, (d, id) => d);

Another way could be using two OR-ed Any based conditions, which would translate to EXISTS(...) OR EXISTS(...) SQL query filter:

query = query.Where(d =>
    db.CarDriversManyToManyTable.Any(cd => d.Id == cd.DriverId && filter.CarIds.Contains(cd.CarId))
    ||
    db.DriverShopManyToManyTable.Any(ds => d.Id == ds.DriverId && filter.ShopIds.Contains(ds.ShopId))
);

You could try and see which one performs better.

Answer 2

The answer to this question is complex and has many facets that, individually, may or may not help in your particular case.

First of all, consider using pagination. .Skip(PageNum * PageSize).Take(PageSize) I doubt your user needs to see millions of rows at once in the front end. Show them only 100, or whatever other smaller number seems reasonable to you.

You've mentioned that you need to use joins to get the data you need. These joins can be done while forming your IQueryable (entity framework), rather than in-memory (linq to objects). Read up on join syntax in linq.

HOWEVER - performing explicit joins in LINQ is not the best practice, especially if you are designing the database yourself. If you are doing database first generation of your entities, consider placing foreign-key constraints on your tables. This will allow database-first entity generation to pick those up and provide you with Navigation Properties which will greatly simplify your code.

If you do not have any control or influence over the database design, however, then I recommend you construct your query in SQL first to see how it performs. Optimize it there until you get the desired performance, and then translate it into an entity framework linq query that uses explicit joins as a last resort.

To speed such queries up, you will likely need to perform indexing on all of the "key" columns that you are joining on. The best way to figure out what indexes you need to improve performance, take the SQL query generated by your EF linq and bring it on over to SQL Server Management Studio. From there, update the generated SQL to provide some predefined values for your @p parameters just to make an example. Once you've done this, right click on the query and either use display estimated execution plan or include actual execution plan. If indexing can improve your query performance, there is a pretty good chance that this feature will tell you about it and even provide you with scripts to create the indexes you need.

Answer 3

It looks to me that using the instance versions of the LINQ extensions is creating several collections before you're done. using the from statement versions should cut that down quite a bit:

driveIds = (from var record in db.CarDriversManyToManyTable
            where filter.CarIds.Contains(record.CarId)
            select record.DriverId).Concat
            (from var record in db.DriverShopManyToManyTable
             where filter.ShopIds.Contains(record.ShopId)
             select record.DriverId).Distinct()

Also using the groupby extension would give better performance than querying each driver Id.