Linq to entity order by sql

Question

I'm doing some tests with EF and Linq to entities to try and improve my application performance.

I just notice something odd (to me) that I can't explain and also can't really tell if incurs considerable overhead.

Here's my linq:

var result = from n in query
        orderby n.PersonId
        select new
        {
            id = n.Id,
            appointmentId = n.AppointmentId,
            message = n.Message,
            wasRead = n.Read,
            canDismiss = (n.Appointment.Status != AppointmentStatus.Waiting),
            date = n.IssueDateUtc
        };

This is the generated sql:

SELECT 
    [Project1].[Id] AS [Id], 
    [Project1].[AppointmentId] AS [AppointmentId], 
    [Project1].[Message] AS [Message], 
    [Project1].[Read] AS [Read], 
    [Project1].[C1] AS [C1], 
    [Project1].[IssueDateUtc] AS [IssueDateUtc]
    FROM ( SELECT 
        [Extent1].[Id] AS [Id], 
        [Extent1].[Read] AS [Read], 
        [Extent1].[Message] AS [Message], 
        [Extent1].[IssueDateUtc] AS [IssueDateUtc], 
        [Extent1].[AppointmentId] AS [AppointmentId], 
        [Extent1].[PersonId] AS [PersonId], 
        CASE WHEN ( NOT ((1 = [Extent2].[Status]) AND ([Extent2].[Status] IS NOT NULL))) THEN cast(1 as bit) WHEN (1 = [Extent2].[Status]) THEN cast(0 as bit) END AS [C1]
        FROM  [dbo].[Notification] AS [Extent1]
        LEFT OUTER JOIN [dbo].[Appointment] AS [Extent2] ON [Extent1].[AppointmentId] = [Extent2].[Id]
        WHERE [Extent1].[PersonId] = @p__linq__0
    )  AS [Project1]
    **ORDER BY [Project1].[PersonId] ASC**

I don't understand the need to kind of group the result in another projection ( Project1 ) whilst this seems to work just fine:

SELECT 
        [Extent1].[Id] AS [Id], 
        [Extent1].[Read] AS [Read], 
        [Extent1].[Message] AS [Message], 
        [Extent1].[IssueDateUtc] AS [IssueDateUtc], 
        [Extent1].[AppointmentId] AS [AppointmentId], 
        [Extent1].[PersonId] AS [PersonId], 
        CASE WHEN ( NOT ((1 = [Extent2].[Status]) AND ([Extent2].[Status] IS NOT NULL))) THEN cast(1 as bit) WHEN (1 = [Extent2].[Status]) THEN cast(0 as bit) END AS [C1]
        FROM  [dbo].[Notification] AS [Extent1]
        LEFT OUTER JOIN [dbo].[Appointment] AS [Extent2] ON [Extent1].[AppointmentId] = [Extent2].[Id]
        WHERE [Extent1].[PersonId] = @p__linq__0
        **ORDER BY [Extent1].[PersonId] ASC**

I found a considerable amount of questionable sql generated by both ef and linq and I'm starting to wonder if I wasn't better off just writing raw sqls.

Question is: is the generated sql extra bits of code something to be worried about? Why is that projection necessary?

Edit to add a new linq

As mentioned in the comments, maybe the verbose was caused by subsequent queries being run. I rewrote the linq to use only one query object, and the result is still the same:

dbSet.Where(n => n.PersonId == id).Select(n => new
            {
                Id = n.Id,
                AppointmentId = n.AppointmentId,
                Message = n.Message,
                Read = n.Read,
                CanBeDismissed = (n.Appointment.Status != AppointmentStatus.Waiting),
                IssueDate = n.IssueDateUtc
            }).OrderBy(n => n.Id).ToList();

Execution plan (same for both sqls )

Edit 2

Just got this query from a simple count.

dbSet.Count(x => x.Id == 1 && x.Read == false);

SELECT 
[GroupBy1].[A1] AS [C1]
FROM ( SELECT 
    COUNT(1) AS [A1]
    FROM [dbo].[Notification] AS [Extent1]
    WHERE ([Extent1].[PersonId] = 19) AND (0 = [Extent1].[Read])
)  AS [GroupBy1]

Expected:

SELECT 
    COUNT(1) AS [A1]
    FROM [dbo].[Notification] AS [Extent1]
    WHERE ([Extent1].[PersonId] = 19) AND (0 = [Extent1].[Read])

I don't get where all this wrappers are coming from and why.

Answer 1

I put together a small sample project on my machine. What acutally causes the projection in your first sample is your conditional calculation of the CanBeDismissed field, resulting in a CASE WHEN in SQL. If you leave this out, Entity Framework won't do an additional projection.

So with an conditional check:

db.Notifications
    .Where(n => n.AppointmentId == 1)
    .OrderBy(n => n.Id)
    .Select(n => new
    {
        Id = n.Id,
        Message = n.Message,
        HasMessage = n.Message != null
    }).ToList();

The SQL produced is:

SELECT 
    [Project1].[Id] AS [Id], 
    [Project1].[Message] AS [Message], 
    [Project1].[C1] AS [C1]
    FROM ( SELECT 
        [Extent1].[Id] AS [Id], 
        [Extent1].[Message] AS [Message], 
        CASE WHEN ([Extent1].[Message] IS NOT NULL) THEN cast(1 as bit) ELSE cast(0 as bit) END AS [C1]
        FROM [dbo].[Notifications] AS [Extent1]
        WHERE 1 = [Extent1].[AppointmentId]
    )  AS [Project1]
    ORDER BY [Project1].[Id] ASC

Let me add the resulting execution plan for later reference:

If you leave it out:

db.Notifications
    .Where(n => n.AppointmentId == 1)
    .OrderBy(n => n.Id)
    .Select(n => new
    {
        Id = n.Id,
        Message = n.Message
    }).ToList();

No projection is done by EF:

SELECT 
    [Extent1].[Id] AS [Id], 
    [Extent1].[Message] AS [Message]
    FROM [dbo].[Notifications] AS [Extent1]
    WHERE 1 = [Extent1].[AppointmentId]
    ORDER BY [Extent1].[Id] ASC

So this is the why. Same applies to your count sample: if there is any grouping happening, EF will add an additional projection which makes the query more verbose. But the important part is, as discussed in the comments to your question, it won't hurt performance , there is no need to worry about this additional projection.

Let me proof this by now adding the execution plan of the following query, where I have just removed the pojection from the first query and moved the orderby to the inner query:

SELECT
[Extent1].[Id] AS [Id], 
[Extent1].[Message] AS [Message], 
CASE WHEN ([Extent1].[Message] IS NOT NULL) THEN cast(1 as bit) ELSE cast(0 as bit) END AS [C1]
FROM [dbo].[Notifications] AS [Extent1]
WHERE 1 = [Extent1].[AppointmentId]
ORDER BY [Extent1].[Id] ASC

It's exactly the same - there is no additional task added and the cost distribution remains the same. The SQL Query Optimizer will optimize such projecitons away just nicely.

So again, don't worry about projections - they won't hurt you, while I agree they seem and are sometimes unnecessarily verbose. But here are two things that might help you:

Performance issues:

First, if you are experiencing performance issues with your query, look at why there happens a Clustered Index Scan in the execution plan you posted. This is not always a sign for some indexing issues, but it is very often. Your problems might root here.

Get rid of unneccesary projections:

If you still want to get rid of those projections in all (or at least more) cases, there is Entity Framework Core 1.0 - it actually produces even nicer SQL than EF 6. It might be worth considering migrating to it, yet be aware that it does not come with all features that EF 6 does, so it might not be an option if you are using features that EF Core 1.0 does not offer. But it will work with the full .NET Framework 4.x!

Here's an example what EF Core 1.0 produces when I execute the first LINQ statement of my answer:

SELECT [n].[Id], [n].[Message], CASE
    WHEN [n].[Message] IS NULL
    THEN CAST(0 AS BIT) ELSE CAST(1 AS BIT)
END
FROM [Notifications] AS [n]
WHERE ([n].[Id] = 1) AND ([n].[Id] = 1)
ORDER BY [n].[Id]