How can I speed up my EF query?

Question

How can I speed up this query? Right now it's taking me around 2 minutes to pull back 210K records.

I turned off LazyLoading as well as set AsNoTracking on my tables.

I know it's a lot of data but surely it shouldn't take 2 minutes to retrieve the data?

context.Configuration.LazyLoadingEnabled = false;

List<MY_DATA> data = context
    .MY_DATA.AsNoTracking()
    .Include(x => x.MY_DATA_DETAILS)
    .Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE) 
             && endDate >= DbFunctions.TruncateTime(x.DB_DATE)
             && x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper())
    .ToList();

Answer 1

You can do without DbFunctions.TruncateTime() and probably also without these Trim().ToUpper() calls.

If you execute a function on a database column before it is filtered, it's impossible for the query optimizer to use any indexes on this column. This is know as being non-sargable . To execute the query in its present form, the database engine has to transform the data first and then scan all the transformed data to do the filtering. No index involved.

DbFunctions.TruncateTime() is meaningless. You have to choose startDate and endDate wisely and use x.DB_DATE as it is.

Further, if x.MY_DATA_DETAILS.CODE is a varchar column (most text columns are), it will be auto-trimmed in searches. Even if the database value contains trailing spaces, they will be ignored. So Trim isn't necessary. Next, most text columns by default have a case-insensitive database collation. You should check it. If this is SQL Server, look for collations like SQL_Latin1_General_CP1_CI_AS . The CI part means Case-Insensitive. If so, you can also do away with the ToUpper part. If not , you can either change the collation to a case-insensitive one, or you maybe should conclude that the column is case sensitive for a reason, so it does matter whether you look for Abc or abc .

Either way, having these transforming function removed form the database columns, the query should be able to run considerably faster, provided that proper indexes are in place.

Answer 2

Usually defining indexes on the columns that you frequently use in the 'where' clause, can improve your performance in selecting the rows from a large tale.

I recommend that you create a stored procedure and move your query into the SP and apply the performance tuning in Database and in your C# code, call the SP.

Answer 3

Do you need all the attributes of the object? You can do this.

    List<MY_DATA> data = context.MY_DATA.Include(x =>
        x.MY_DATA_DETAILS).Where(x => startDate <= DbFunctions.TruncateTime(x.DB_DATE) &&
        endDate >= DbFunctions.TruncateTime(x.DB_DATE) &&
        x.MY_DATA_DETAILS.CODE.Trim().ToUpper() == myCode.Trim().ToUpper()).select(x => new MY_DATA()
        {
            Value = data
        }).ToList();

Answer 4

In addition to what the other guy said about moving your query to a stored proc and creating the proper indexes, I'd say you'd be better off using SQL reporting rather then trying to import the data into your application and reporting from there. Especially 210K rows. SQL has internal optimizations that you'll never be able to come close to with stored procs and queries.

You can see this very easily:

1) try write a simple console app that tries to pull down that entire table and writes it to a CSV file -- it'll be extremely slow.

2) try use the data export through Sql Mgmt Studio and export to a CSV -- it'll be done in a few seconds.