EF Core Include() statement is null for IQueryable

Question

Ok, so this might be a bit hard to explain without a ton of code to support it but I will try my best.

Essentially I am doing a query (currently on ef core 2.1) with involves a 1 to many relationships. However, the "many" collection is null when it materialized.

Here is the query in question (some code removed for brevity)

IQueryable<AccountViewModel> baseQuery = from ms in _managedSupportRepository.GetAllIncluding(m => m.Users) // here is the problem
                                          // a few lines of filters like the one below
                                          where string.IsNullOrEmpty(clientVersionFilter) || !string.IsNullOrEmpty(ms.ClientVersion) && ms.ClientVersion.Contains(clientVersionFilter, StringComparison.OrdinalIgnoreCase)
                                          join c in _contractRepository.GetAll() on ms.Id equals c.AssetId into contracts
                                          from c in contracts.DefaultIfEmpty()
                                          let isAssigned = c != null
                                          where !isAssignedFilter.valueExists || isAssignedFilter.value == isAssigned
                                          join a in _autotaskAccountRepository.GetAll() on ms.TenantId equals a.Id
                                          where string.IsNullOrEmpty(accountNameFilter) || !string.IsNullOrEmpty(a.AccountName) && a.AccountName.Contains(accountNameFilter, StringComparison.OrdinalIgnoreCase)
                                          select new AccountViewModel
                                          {
                                              AccountName = a.AccountName,
                                              ActiveUsers = ms.GetConsumed(), // here is the problem
                                              ClientVersion = ms.ClientVersion,
                                              ExternalIpAddress = ms.IpAddress,
                                              Hostname = ms.Hostname,
                                              Id = ms.Id,
                                              IsActive = ms.IsActive,
                                              IsAssigned = isAssigned,
                                              LastSeen = ms.CheckInTime,
                                              Status = ms.Status
                                          };

int count = baseQuery.Count();

baseQuery = baseQuery.Paging(sortOrder, start, length);

return (baseQuery.ToList(), count);

Just for clarity, the _managedSupportRepository.GetAllIncluding(m => m.Users) method is just a wrapper around the .Include() method.

So the problem is in the view model for active users ActiveUsers = ms.GetConsumed(), . The GetConsumed() method is as follows

public long GetConsumed()
{
    return Users.Count(u => !u.IsDeleted && u.Enabled && u.UserType == UserType.Active);
}

however, this throws a null reference exception because the Users collection is null. Now my question is, why is the Users collection null when I am explicitly asking it to be loaded? A workaround at the moment is to alter the queries first line to be this _managedSupportRepository.GetAllIncluding(m => m.Users).AsEnumerable() which is just ridiculous as it brings all the records back (which is several thousand) so performance is nonexistent.

The reason it needs to be an IQueryable is so the paging can be applied, thus reducing the amount of information pulled from the database.

Any help is appreciated.

Answer 1

There are two parts to this problem:

1) Includes not in the projection don't get included

When you do queries on EF on the provider (server evaluation), you are not -executing- your new expressions, so this:

ActiveUsers = ms.GetConsumed(),

Never actually executes ms.GetConsumed() . The expression you pass in for the new is parsed and then translated to the query (SQL in case of sql server), but ms.GetConsumed() is not executed on the provider (on the query to the database).

So you need to include Users on the expression . For example:

select new AccountViewModel
{
    AccountName = a.AccountName,
    AllUsers = Users.ToList(),
    ActiveUsers = ms.GetConsumed(),
    // etc.
}

This way EF knows it needs Users for the query and actually includes it (you are not using Users in your expression, so EF thinks it doesn't need it even if you Include() it... it'll probably show a warning on the Output window in Visual Studio), otherwise it tries to project and request only the fields it understands from the new expression (which doesn't include Users ).

So you need to be explicit here... try:

ActiveUsers = Users.Count(u => !u.IsDeleted && u.Enabled && u.UserType == UserType.Active);

And Users will be actually included.

2) Automatic client side evaluation when queries can't get translated

In this case, Users will be included because it's in the actual expression... BUT , EF still doesn't know how to translate ms.GetConsumed() to the provider query, so it'll work (because Users will be loaded), but it won't be ran on the database, it'll still be run on memory (it'll will do client side projection). Again, you should see a warning about this in the Output window in Visual Studio if you are running it there.

EF Core allows this (EF6 didn't), but you can configure it to throw errors if this happens (queries that get evaluated in memory):

protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder)
{
    optionsBuilder
        /* etc. */
        .ConfigureWarnings(warnings => warnings.Throw(RelationalEventId.QueryClientEvaluationWarning));
}

You can read more about this here: https://docs.microsoft.com/en-us/ef/core/querying/client-eval