LINQ does not produce COUNT() query on group using EF Core

Question

I have a ASP.NET Core application which works on an SQL Server 2012 via EF Core. What I want to achieve is counting the size of different groups like

from b in this._context.Benchmark
group b by b.Device into g
select new {
    Device = g.Key,
    Count = g.Count()
}

The problem is that the whole thing is extremely slow, and the reason for that seems to be that the SQL statement does not map to a GROUP BY and COUNT(*) , but according to the debugger, the application requests everything and performs the counting on the CPU. The query I get from the debugger's event window looks like:

SELECT [b0].[ID], [b0].[CreateDate], [b0].[Creator], [b0].[Device], [b0].[Environment], [b0].[Machine], [b0].[Name], [b0].[Plugin], [b0].[RunDate]
FROM [Benchmark] AS [b0]
ORDER BY [b0].[Device]

I can also see that the memory consumption of the IIS Express I use for debugging is insane for a simple COUNT() , so I tend to believe that this is the actual query.

The question is: how can i rephrase the query such that it actually maps to a COUNT() ?

Edit : I have tried the very same query on the "real" EF and ctx.Database.Log = Console.Write , which produces the expected COUNT() query, which makes me believe this is an issue of EF Core.

Answer 1

请将EF Core升级到2.1.2版，现在支持。

Answer 2

The LINQ expression 'GroupBy([x.Device], [x])' could not be translated and will be evaluated locally.

The LINQ expression 'Count()' could not be translated and will be evaluated locally.

The above logs are from my later test results, which show that GroupBy & Count() are not supported yet to translate full LINQ to SQL, then directly execute full SQL in database, now EF Core has to fetch all data first, then GroupBy results locally. So, if your data is huge, then the performance will be very bad.

You have to hard code a simple SQL to do your work in database directly, then the performance will be back to normal.

I upgraded to EF Core 2.1.2, tried the following code both in MySQL and MS SQL :

var query = _context.Benchmark.GroupBy(x => x.Device, 
              (key, group) => new  //Result selector
              {
                  Device = key,
                  Count = group.Count()
              });
await query.ToListAsync();

The result selector is right inside GroupBy which will make sure whether it could be translated to right SQL, bit unfortunately, it's NOT.

Here is another tricky, avoid to use Count :

var query = _context.Benchmark.GroupBy(x => x.Device, 
              (key, group) => new  //Result selector
              {
                  Device = key,
                  Count = group.Select(g => g.Id)//Avoid to use Count
              });
await query.ToListAsync();

The C# code produces the following logs by this tool :

2018-08-24T14:27:26.6737424-04:00 Information 10403 Microsoft.EntityFrameworkCore.Infrastructure
Entity Framework Core 2.1.2-rtm-30932 initialized 'ApplicationDbContext' using provider 'Pomelo.EntityFrameworkCore.MySql' with options: None
2018-08-24T14:27:31.4270317-04:00 Debug 10101 Microsoft.EntityFrameworkCore.Query
Compiling query model: 
'from IGrouping<Device, Benchmark> <generated>_0 in 
    (from Benchmark x in DbSet<Benchmark>
    select [x]).GroupBy([x].Device, [x])
select new <>f__AnonymousType3<Device, int>(
    [<generated>_0].Key, 

        (from Benchmark <generated>_1 in [<generated>_0]
        select [<generated>_1]).Count()
)'
2018-08-24T14:27:31.4319437-04:00 Debug 10104 Microsoft.EntityFrameworkCore.Query
Optimized query model: 
'from IGrouping<Device, Benchmark> <generated>_0 in 
    (from Benchmark x in DbSet<Benchmark>
    join Device x.Device in DbSet<Device>
    on Property([x], "DeviceId") equals Property([x.Device], "Id")
    select [x]).GroupBy([x.Device], [x])
select new <>f__AnonymousType3<Device, int>(
    [<generated>_0].Key, 

        (from Benchmark <generated>_1 in [<generated>_0]
        select [<generated>_1]).Count()
)'

Here is core LOGS, which shows the GROUPBY is not supported yet:

 2018-08-24T14:27:31.4431635-04:00 Warning 20500 Microsoft.EntityFrameworkCore.Query The LINQ expression 'GroupBy([x.Device], [x])' could not be translated and will be evaluated locally. 2018-08-24T14:27:31.4476637-04:00 Warning 20500 Microsoft.EntityFrameworkCore.Query The LINQ expression 'Count()' could not be translated and will be evaluated locally. 2018-08-24T14:27:31.4511652-04:00 Warning 20500 Microsoft.EntityFrameworkCore.Query The LINQ expression 'Count()' could not be translated and will be evaluated locally. 

Here is rest logs after:

2018-08-24T14:27:31.4608060-04:00 Debug 10107 Microsoft.EntityFrameworkCore.Query
(QueryContext queryContext) => IAsyncEnumerable<<>f__AnonymousType3<Device, int>> _InterceptExceptions(
    source: IAsyncEnumerable<<>f__AnonymousType3<Device, int>> _SelectAsync(
        source: IAsyncEnumerable<IGrouping<Device, ValueBuffer>> _GroupBy(
            source: IAsyncEnumerable<TransparentIdentifier<ValueBuffer, Device>> _ShapedQuery(
                queryContext: queryContext, 
                shaperCommandContext: SelectExpression: 
                    SELECT `x.Device`.`Id`, `x.Device`.`Description`, `x.Device`.`IP`, `x.Device`.`Name`, `x.Device`.`Port`
                    FROM `Benchmark` AS `x`
                    INNER JOIN `Device` AS `x.Device` ON `x`.`DeviceId` = `x.Device`.`Id`
                    ORDER BY `x.Device`.`Id`, 
                shaper: TypedCompositeShaper<ValueBufferShaper, ValueBuffer, BufferedOffsetEntityShaper<Device>, Device, TransparentIdentifier<ValueBuffer, Device>>), 
            keySelector: (TransparentIdentifier<ValueBuffer, Device> t0) => t0.Inner, 
            elementSelector: (TransparentIdentifier<ValueBuffer, Device> t0) => t0.Outer), 
        selector: (IGrouping<Device, ValueBuffer> <generated>_0 | CancellationToken ct) => Task<<>f__AnonymousType3<Device, int>> _ExecuteAsync(
            taskFactories: new Func<Task<object>>[]{ () => Task<object> _ToObjectTask(Task<int> Count(
                        source: IAsyncEnumerable<ValueBuffer> _ToAsyncEnumerable(<generated>_0), 
                        cancellationToken: queryContext.CancellationToken)) }, 
            selector: (object[] results) => new <>f__AnonymousType3<Device, int>(
                <generated>_0.Key, 
                (int)results[0]
            ))), 
    contextType: Sequencer.Updater.Data.ApplicationDbContext, 
    logger: DiagnosticsLogger<Query>, 
    queryContext: queryContext)
2018-08-24T14:27:31.4825759-04:00 Debug 20000 Microsoft.EntityFrameworkCore.Database.Connection
Opening connection to database 'TestDB' on server 'localhost,3306'.
2018-08-24T14:27:31.4877302-04:00 Debug 20001 Microsoft.EntityFrameworkCore.Database.Connection
Opened connection to database 'TestDB' on server 'localhost,3306'.
2018-08-24T14:27:31.4901269-04:00 Debug 20100 Microsoft.EntityFrameworkCore.Database.Command
Executing DbCommand [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT `x.Device`.`Id`, `x.Device`.`Description`, `x.Device`.`IP`, `x.Device`.`Name`, `x.Device`.`Port`
FROM `Benchmark` AS `x`
INNER JOIN `Device` AS `x.Device` ON `x`.`DeviceId` = `x.Device`.`Id`
ORDER BY `x.Device`.`Id`
2018-08-24T14:27:31.4929857-04:00 Information 20101 Microsoft.EntityFrameworkCore.Database.Command
Executed DbCommand (3ms) [Parameters=[], CommandType='Text', CommandTimeout='30']
SELECT `x.Device`.`Id`, `x.Device`.`Description`, `x.Device`.`IP`, `x.Device`.`Name`, `x.Device`.`Port`
FROM `Benchmark` AS `x`
INNER JOIN `Device` AS `x.Device` ON `x`.`DeviceId` = `x.Device`.`Id`
ORDER BY `x.Device`.`Id`
2018-08-24T14:27:31.5231128-04:00 Debug 20300 Microsoft.EntityFrameworkCore.Database.Command
A data reader was disposed.
2018-08-24T14:27:31.5270399-04:00 Debug 20002 Microsoft.EntityFrameworkCore.Database.Connection
Closing connection to database 'TestDB' on server 'localhost,3306'.
2018-08-24T14:27:31.5303748-04:00 Debug 20003 Microsoft.EntityFrameworkCore.Database.Connection
Closed connection to database 'TestDB' on server 'localhost,3306'.

Here is my environment:

PM> dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.401
 Commit:    91b1c13032

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.401\

Host (useful for support):
  Version: 2.1.3
  Commit:  124038c13e

.NET Core SDKs installed:
  2.1.202 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.401 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 2.0.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.3 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]