Entity Framework Core Dynamic GroupBy

Question

Instead of hard-coding a GroupBy clause, I would like to be able to dynamically supply a GroupBy clause to an EF query.

Hopefully this code illustrates what I am trying to do:

Here are some simple entities for a simple library book loan system:

public class Book
{
   public int Id { get; set; }
   public string Isbn { get; set; }
   public string Title { get; set; }

   public IList<Loan> Loans { get; set; }
}


public class Member
{
   public int Id { get; set; }
   public string FirstName { get; set; }
   public string Surname { get; set; }
   public int Age { get; set; }

   public IList<Loan> Loans { get; set; }
}


public class Loan
{
   public int Id { get; set; }
   public int BookId { get; set; }
   public int MemberId { get; set; }
   public DateTime StartDate { get; set; }
   public DateTime EndDate { get; set; }

   public Book Book { get; set; }
   public Member Member { get; set; }
}

I have these two view models to help with my EF queries:

public class DoubleGroupByClause
{
   public string GroupByFieldValue1 { get; set; }
   public string GroupByFieldValue2 { get; set; }
}

public class QueryRow
{
   public string GroupBy1 { get; set; }
   public string GroupBy2 { get; set; }
   public int Value { get; set; }
}

I can run the following code, and it will return a list of QueryRow objects, which equates to the total number of Loans, grouped by book and member age:

var rows = _db.Loans
   // note that the GroupBy method params are hard-coded
   .GroupBy( x => new DoubleGroupByClause{GroupByFieldValue1 = x.BookId, GroupByFieldValue2 = x.Member.Age} )
   .Select(x => new QueryRow
   {
      RowId = x.Key.GroupByFieldValue1.ToString(),
      ColId = x.Key.GroupByFieldValue2.ToString(),
      Value = x.Count()
   })
   .ToList();

This works just fine, and returns me a list similar to the following:

GroupBy1   GroupBy2   Value
(BookId)   (Age)      (Loans)
========   ========   =====
      45         14      23
      45         15      37
      45         16      55
      72         14      34
      72         15      66
      72         16       9

Now on to the problem: I want to be able to specify my GroupBy clause from outside of the query itself. The idea is that the two fields that I wish to group by should be held in something similar to variables, and then applied to the query at run-time. I am very close to doing it. Here is where I am up to:

// holding the two group by clauses here
Func<Loan, string> groupBy1 = g => g.BookId.ToString();
Func<Loan, string> groupBy2 = g => g.Member.Age.ToString();

// applying the clauses into an expression
Expression<Func<Loan, DoubleGroupByClause>> groupBy = g => new DoubleGroupByClause {GroupByFieldValue1 = groupBy1.Invoke(g), GroupByFieldValue2 = groupBy2.Invoke(g)};

var rows = _db.Loans
   .GroupBy(groupBy)        // applying the expression into the GroupBy clause  
   .Select(x => new QueryRow
   {
      RowId = x.Key.GroupByFieldValue1.ToString(),
      ColId = x.Key.GroupByFieldValue2.ToString(),
      Value = x.Count()
   })
   .ToList();

This compiles just fine, but I get the following error at runtime:

System.InvalidOperationException: The LINQ expression 'DbSet<Loans>
    .GroupBy(
        source: m => new DoubleGroupByClause{
            GroupByFieldValue1 = __groupBy1_0.Invoke(m),
            GroupByFieldValue2 = __groupBy2_1.Invoke(m)
        }
        ,
        keySelector: m => m)' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().

I think I'm quite close, but can anyone help me with how I can get this working?

There may be a much simpler way of doing this, so I am open to any suggestions. Hopefully the code illustrates what I am trying to achieve though.

Answer 1

The arguments should be expressions ( Expression<Func<>> ) rather than delegates ( Func<> ), eg

Expression<Func<Loan, string>> groupBy1 = g => g.BookId.ToString();
Expression<Func<Loan, string>> groupBy2 = g => g.Member.Age.ToString();

Then you need to compose Expression<Func<Loan, DoubleGroupByClause>> from them, which is not so natural as with delegates, but still possible with the help of the Expression class and the following little helper utility class for replacing lambda expression parameters:

public static partial class ExpressionUtils
{
    public static Expression ReplaceParameter(this Expression expression, ParameterExpression source, Expression target)
        => new ParameterReplacer { source = source, target = target }.Visit(expression);

    class ParameterReplacer : ExpressionVisitor
    {
        public ParameterExpression source;
        public Expression target;
        protected override Expression VisitParameter(ParameterExpression node)
            => node == source ? target : node;
    }
}

Now the implementation of the desired expression:

Expression<Func<string, string, DoubleGroupByClause>> groupByPrototype = (v1, v2) =>
    new DoubleGroupByClause { GroupByFieldValue1 = v1, GroupByFieldValue2 = v2 };
var parameter = groupBy1.Parameters[0];
var v1 = groupBy1.Body;
var v2 = groupBy2.Body.ReplaceParameter(groupBy2.Parameters[0], parameter);
var body = groupByPrototype.Body
    .ReplaceParameter(groupByPrototype.Parameters[0], v1)
    .ReplaceParameter(groupByPrototype.Parameters[1], v2);
var groupBy = Expression.Lambda<Func<Loan, DoubleGroupByClause>>(body, parameter);

Answer 2

The actual solution was frightfully close to my attempt, so I'm posting in here in case it helps anyone else.

All that was missing was a call to.AsExpandable(). This seems to pre-evaluate the Expressions, and then everything works as expected.

So in my example above, it just needed to look like this:

var rows = _db.Loans
   .AsExpandable()
   .GroupBy(groupBy)        // applying the expression into the GroupBy clause  
   .Select(x => new QueryRow
   {
      RowId = x.Key.GroupByFieldValue1.ToString(),
      ColId = x.Key.GroupByFieldValue2.ToString(),
      Value = x.Count()
   })
   .ToList();

Thanks to Ivan Stoev for your initial suggestions though. That led me on the correct path to help me eventually solve it.