EF Core - Filtering by a child collection of Key Value pairs extremely slow

Question

My main system entity is 'tagged' with a child collection of key value pairs, which I want to use to filter a listing of the main entities. However, the EF core query I've written below is far too slow for acceptable use.

Simplified Entity Classes

 public class MainEntity
 {
    public int Id { get; set; }
    public DateTimeOffset Created { get; set; }
    public string Stuff {get; set;}
    public virtual List<Tag> Tags { get; set; }
 }

 public class Tag
 {
    public int Id { get; set; }
    public string Key { get; set; }
    public string Value { get; set; }

    public int MainEntityId { get; set; }
    public virtual MainEntity MainEntity { get; set; }
 }

Simplified Query

//filter params passed into the query function
//String? stuffFilter
//List<Tag> tagSearchValues

var query = _dbContext.MainEntities.Where(
    me => ((!stuffFilter.HasValue || me.Stuff == stuffFilter.Value)                    
    && (tagSearchValues == null || tagSearchValues.Count == 0 ||
    (
    (me.Tags.Select(t => t.Key).Any(tk => tagSearchValues.Select(s => s.Key).Any(sk => sk == tk))) &&
    (me.Tags.Select(t => t.Value).Any(tv => tagSearchValues.Select(s => s.Value).Any(sv => sv == tv)))
    )
    ).                    
    OrderByDescending(l => me.Created).AsNoTracking();

I'm a bit rusty with EF (and using EF Core for the first time), but the problem is down to the way I'm filtering by the child Tag collection with the multiple .Any() commands (the query performs perfectly when no Tag filters are specified).

I can't think of a another way to filter the child Tag object collection against the selected Tag filter objects - a single filter Tag would be much simpler, and quicker, I imagine.

The only alternative I can currently think of is to do a custom SQL query myself, but it seems a shame to resort to that already when putting together my first EF Core query!

Answer 1

The first thing to note is that your proposed query can't be evaluated fully as SQL because there's no SQL equivalent for a collection containing non-primitive values, tagSearchValues . This causes EF to auto-switch to client-side evaluation . That is, it pulls into memory all entities that meet the stuffFilter condition and all of their tags, and then it applies the tags predicate. That, evidently, is not efficient.

Secondly, the query is inaccurate. Entities containing tags with certain keys and tags with certain values isn't the same as tags containing specific key/value combinations . It requires a query that matches each combination, like this:

db.MainEntities.Where(...)
    .Where(m => tagSearchValues
       .Any(t => m.Tags.Any(mt => mt.Key == t.Key 
                               && mt.Value == t.Value)))

However, if you do that, EF will again turn to inefficient client-side evaluation and you'd even have to apply Include or lazy loading yourself to pull the tags into memory. (Moreover, for some reason, EF fires tons of redundant queries).

Fact of the matter is that EF (like other ORMs) isn't well-geared to such pair-wise comparisons server-side. Therefore you need a predicate builder to build the tag predicates. There are several predicate buiders, fe in Linqkit . I use this one because it's nice and simple. The recipe is: build a predicate and apply it in a Where() :

var tagPredicate = PredicateBuilder.True<MainEntity>();
if (tagSearchValues.Any())
{
    tagPredicate = PredicateBuilder.False<MainEntity>();
    foreach (var tag in tagSearchValues)
    {
        tagPredicate = tagPredicate.Or(m => m.Tags
                           .Any(t => t.Key == tag.Key
                                  && t.Value == tag.Value));
    }
}

var query = _dbContext.MainEntities
    .Where(m => string.IsNullOrWhiteSpace(stuff) || m.Stuff == stuff)
    .Where(tagPredicate);
... // Use query

I use Or because I assume (from your query) that you want entities having any tag in the search tags. That's why I start with a PredicateBuilder.True predicate, so the query will return results if there are no search tags, similar to your original query.

Answer 2

Do you know what SQL is being generated by EF Core Any ? EF Core has the unfortunate design property of silently executing queries on the client side if they can't be translated to SQL.

What if you consolidate the Key and Value testing?

(me.Tags.Any(met => tagSearchValues.Any(st => st.Tag == met.Tag && st.Value == met.Value)))

Or what if you use Contains instead?

(me.Tags.Select(t => t.Key).Any(tk => tagSearchValues.Select(s => s.Key).Contains(tk))) &&
(me.Tags.Select(t => t.Value).Any(tv => tagSearchValues.Select(s => s.Value).Contains(tv)))