Cleaning a big sql query generated by the Entity Framework

Question

I was checking my SQL queries to see if they could be optimized and then I found a massive query which only contains little logic.

I understand some parts of the query but I am not sure why it selects so many columns each time.

Why is Entity Framework assigning / selecting the website entity so much? (The website entity contains columns like: id, title, description).

 FROM      (SELECT 
                    [Project4].[Id] AS [Id], 
                    [Project4].[Title] AS [Title], 
                    [Project4].[Description] AS [Description], 
                    [Project4].[Url] AS [Url], 
                    [Project4].[BannerURL] AS [BannerURL], 
                    [Project4].[UserID] AS [UserID], 
                    [Project4].[CategoryID] AS [CategoryID], 
                    [Project4].[Keywords] AS [Keywords], 
                    [Project4].[Enabled] AS [Enabled], 
                    [Project4].[DateAdded] AS [DateAdded], 
                    [Project4].[Sponsored] AS [Sponsored], 
                    [Project4].[ServerIP] AS [ServerIP], 
                    [Project4].[ServerPort] AS [ServerPort], 
                    [Project4].[MonitorCheckedDate] AS [MonitorCheckedDate], 
                    [Project4].[IsOnline] AS [IsOnline], 
                    [Project4].[BannerFileName] AS [BannerFileName], 
                    [Project4].[Thumbnail] AS [Thumbnail], 
                    [Project4].[C1] AS [C1], 
                    [Project4].[C2] AS [C2], 

Full sql query:

SELECT TOP (25) 
    [Filter8].[Id1] AS [Id], 
    [Filter8].[Title] AS [Title], 
    [Filter8].[Description] AS [Description], 
    [Filter8].[Url] AS [Url], 
    [Filter8].[BannerURL] AS [BannerURL], 
    [Filter8].[UserID] AS [UserID], 
    [Filter8].[CategoryID] AS [CategoryID], 
    [Filter8].[Keywords] AS [Keywords], 
    [Filter8].[Enabled] AS [Enabled], 
    [Filter8].[DateAdded] AS [DateAdded], 
    [Filter8].[Sponsored] AS [Sponsored], 
    [Filter8].[ServerIP] AS [ServerIP], 
    [Filter8].[ServerPort] AS [ServerPort], 
    [Filter8].[MonitorCheckedDate] AS [MonitorCheckedDate], 
    [Filter8].[IsOnline] AS [IsOnline], 
    [Filter8].[BannerFileName] AS [BannerFileName], 
    [Filter8].[Thumbnail] AS [Thumbnail]

    // What's going on here?

    FROM ( SELECT [Filter7].[Id1], [Filter7].[Title], [Filter7].[Description], [Filter7].[Url], [Filter7].[BannerURL], [Filter7].[UserID], [Filter7].[CategoryID], [Filter7].[Keywords], [Filter7].[Enabled], [Filter7].[DateAdded], [Filter7].[Sponsored], [Filter7].[ServerIP], [Filter7].[ServerPort], [Filter7].[MonitorCheckedDate], [Filter7].[IsOnline], [Filter7].[BannerFileName], [Filter7].[Thumbnail], [Filter7].[C1], [Filter7].[C2], [Filter7].[C3], [Filter7].[IsAdminVerified1], [Filter7].[IsEmailVerified1], [Filter7].[BannedEndDate1], [Filter7].[BannedEndDate2], row_number() OVER (ORDER BY [Filter7].[Sponsored] DESC, [Filter7].[C2] DESC, [Filter7].[C3] DESC, [Filter7].[C1] DESC, [Filter7].[DateAdded] DESC) AS [row_number]
        FROM ( SELECT [Filter6].[Id1], [Filter6].[Title], [Filter6].[Description], [Filter6].[Url], [Filter6].[BannerURL], [Filter6].[UserID], [Filter6].[CategoryID], [Filter6].[Keywords], [Filter6].[Enabled], [Filter6].[DateAdded], [Filter6].[Sponsored], [Filter6].[ServerIP], [Filter6].[ServerPort], [Filter6].[MonitorCheckedDate], [Filter6].[IsOnline], [Filter6].[BannerFileName], [Filter6].[Thumbnail], [Filter6].[C1], [Filter6].[C2], [Filter6].[C3], [Filter6].[IsAdminVerified1], [Filter6].[IsEmailVerified1], [Filter6].[BannedEndDate1], [Filter6].[BannedEndDate2]
            FROM ( SELECT [Filter5].[Id1], [Filter5].[Title], [Filter5].[Description], [Filter5].[Url], [Filter5].[BannerURL], [Filter5].[UserID], [Filter5].[CategoryID], [Filter5].[Keywords], [Filter5].[Enabled], [Filter5].[DateAdded], [Filter5].[Sponsored], [Filter5].[ServerIP], [Filter5].[ServerPort], [Filter5].[MonitorCheckedDate], [Filter5].[IsOnline], [Filter5].[BannerFileName], [Filter5].[Thumbnail], [Filter5].[C1], [Filter5].[C2], [Filter5].[C3], [Filter5].[IsAdminVerified1], [Filter5].[IsEmailVerified1], [Filter5].[BannedEndDate1], [Filter5].[BannedEndDate2]
                FROM ( SELECT [Project5].[Id] AS [Id1], [Project5].[Title] AS [Title], [Project5].[Description] AS [Description], [Project5].[Url] AS [Url], [Project5].[BannerURL] AS [BannerURL], [Project5].[UserID] AS [UserID], [Project5].[CategoryID] AS [CategoryID], [Project5].[Keywords] AS [Keywords], [Project5].[Enabled] AS [Enabled], [Project5].[DateAdded] AS [DateAdded], [Project5].[Sponsored] AS [Sponsored], [Project5].[ServerIP] AS [ServerIP], [Project5].[ServerPort] AS [ServerPort], [Project5].[MonitorCheckedDate] AS [MonitorCheckedDate], [Project5].[IsOnline] AS [IsOnline], [Project5].[BannerFileName] AS [BannerFileName], [Project5].[Thumbnail] AS [Thumbnail], [Project5].[C1] AS [C1], [Project5].[C2] AS [C2], [Project5].[C3] AS [C3], [Extent6].[IsAdminVerified] AS [IsAdminVerified1], [Extent7].[IsEmailVerified] AS [IsEmailVerified1], [Extent8].[BannedEndDate] AS [BannedEndDate1], [Extent9].[BannedEndDate] AS [BannedEndDate2]

                    // Why the repeat?

                    FROM      (SELECT 
                        [Project4].[Id] AS [Id], 
                        [Project4].[Title] AS [Title], 
                        [Project4].[Description] AS [Description], 
                        [Project4].[Url] AS [Url], 
                        [Project4].[BannerURL] AS [BannerURL], 
                        [Project4].[UserID] AS [UserID], 
                        [Project4].[CategoryID] AS [CategoryID], 
                        [Project4].[Keywords] AS [Keywords], 
                        [Project4].[Enabled] AS [Enabled], 
                        [Project4].[DateAdded] AS [DateAdded], 
                        [Project4].[Sponsored] AS [Sponsored], 
                        [Project4].[ServerIP] AS [ServerIP], 
                        [Project4].[ServerPort] AS [ServerPort], 
                        [Project4].[MonitorCheckedDate] AS [MonitorCheckedDate], 
                        [Project4].[IsOnline] AS [IsOnline], 
                        [Project4].[BannerFileName] AS [BannerFileName], 
                        [Project4].[Thumbnail] AS [Thumbnail], 
                        [Project4].[C1] AS [C1], 
                        [Project4].[C2] AS [C2], 
                        (SELECT 
                            COUNT(1) AS [A1]
                            FROM [dbo].[WebsiteOut] AS [Extent5]
                            WHERE ([Project4].[Id] = [Extent5].[WebsiteID]) AND ([Extent5].[Unique] = 1)) AS [C3]
                        FROM ( SELECT 
                            [Project2].[Id] AS [Id], 
                            [Project2].[Title] AS [Title], 
                            [Project2].[Description] AS [Description], 
                            [Project2].[Url] AS [Url], 
                            [Project2].[BannerURL] AS [BannerURL], 
                            [Project2].[UserID] AS [UserID], 
                            [Project2].[CategoryID] AS [CategoryID], 
                            [Project2].[Keywords] AS [Keywords], 
                            [Project2].[Enabled] AS [Enabled], 
                            [Project2].[DateAdded] AS [DateAdded], 
                            [Project2].[Sponsored] AS [Sponsored], 
                            [Project2].[ServerIP] AS [ServerIP], 
                            [Project2].[ServerPort] AS [ServerPort], 
                            [Project2].[MonitorCheckedDate] AS [MonitorCheckedDate], 
                            [Project2].[IsOnline] AS [IsOnline], 
                            [Project2].[BannerFileName] AS [BannerFileName], 
                            [Project2].[Thumbnail] AS [Thumbnail], 
                            CASE WHEN ( EXISTS (SELECT 
                                1 AS [C1]
                                FROM [dbo].[WebsiteRating] AS [Extent4]
                                WHERE [Project2].[Id] = [Extent4].[WebsiteID]
                            )) THEN  CAST( [Project2].[C2] AS int) ELSE 5 END AS [C1], 
                            [Project2].[C1] AS [C2]
                            FROM ( SELECT 
                                [Project1].[Id] AS [Id], 
                                [Project1].[Title] AS [Title], 
                                [Project1].[Description] AS [Description], 
                                [Project1].[Url] AS [Url], 
                                [Project1].[BannerURL] AS [BannerURL], 
                                [Project1].[UserID] AS [UserID], 
                                [Project1].[CategoryID] AS [CategoryID], 
                                [Project1].[Keywords] AS [Keywords], 
                                [Project1].[Enabled] AS [Enabled], 
                                [Project1].[DateAdded] AS [DateAdded], 
                                [Project1].[Sponsored] AS [Sponsored], 
                                [Project1].[ServerIP] AS [ServerIP], 
                                [Project1].[ServerPort] AS [ServerPort], 
                                [Project1].[MonitorCheckedDate] AS [MonitorCheckedDate], 
                                [Project1].[IsOnline] AS [IsOnline], 
                                [Project1].[BannerFileName] AS [BannerFileName], 
                                [Project1].[Thumbnail] AS [Thumbnail], 
                                [Project1].[C1] AS [C1], 
                                (SELECT 
                                    AVG( CAST( [Extent3].[Rating] AS float)) AS [A1]
                                    FROM [dbo].[WebsiteRating] AS [Extent3]
                                    WHERE [Project1].[Id] = [Extent3].[WebsiteID]) AS [C2]
                                FROM ( SELECT 
                                    [Extent1].[Id] AS [Id], 
                                    [Extent1].[Title] AS [Title], 
                                    [Extent1].[Description] AS [Description], 
                                    [Extent1].[Url] AS [Url], 
                                    [Extent1].[BannerURL] AS [BannerURL], 
                                    [Extent1].[UserID] AS [UserID], 
                                    [Extent1].[CategoryID] AS [CategoryID], 
                                    [Extent1].[Keywords] AS [Keywords], 
                                    [Extent1].[Enabled] AS [Enabled], 
                                    [Extent1].[DateAdded] AS [DateAdded], 
                                    [Extent1].[Sponsored] AS [Sponsored], 
                                    [Extent1].[ServerIP] AS [ServerIP], 
                                    [Extent1].[ServerPort] AS [ServerPort], 
                                    [Extent1].[MonitorCheckedDate] AS [MonitorCheckedDate], 
                                    [Extent1].[IsOnline] AS [IsOnline], 
                                    [Extent1].[BannerFileName] AS [BannerFileName], 
                                    [Extent1].[Thumbnail] AS [Thumbnail], 
                                    (SELECT 
                                        COUNT(1) AS [A1]
                                        FROM [dbo].[WebsiteIn] AS [Extent2]
                                        WHERE ([Extent1].[Id] = [Extent2].[WebsiteID]) AND ([Extent2].[Unique] = 1)) AS [C1]
                                    FROM [dbo].[Websites] AS [Extent1]
                                )  AS [Project1]
                            )  AS [Project2]
                        )  AS [Project4] ) AS [Project5]
                    INNER JOIN [dbo].[Users] AS [Extent6] ON [Project5].[UserID] = [Extent6].[Id]
                    INNER JOIN [dbo].[Users] AS [Extent7] ON [Project5].[UserID] = [Extent7].[Id]
                    LEFT OUTER JOIN [dbo].[Users] AS [Extent8] ON [Project5].[UserID] = [Extent8].[Id]
                    LEFT OUTER JOIN [dbo].[Users] AS [Extent9] ON [Project5].[UserID] = [Extent9].[Id]
                    WHERE [Project5].[Enabled] = 1
                )  AS [Filter5]
                WHERE [Filter5].[IsAdminVerified1] = 1
            )  AS [Filter6]
            WHERE [Filter6].[IsEmailVerified1] = 1
        )  AS [Filter7]
        WHERE ([Filter7].[BannedEndDate1] IS NULL) OR ( CAST( SysDateTime() AS datetime2) > [Filter7].[BannedEndDate2])
    )  AS [Filter8]
    WHERE [Filter8].[row_number] > 0
    ORDER BY [Filter8].[Sponsored] DESC, [Filter8].[C2] DESC, [Filter8].[C3] DESC, [Filter8].[C1] DESC, [Filter8].[DateAdded] DESC

Linq:

(from website in _unitOfWorkRepository.WebsitesRepository.GetAll()
               let amountVotes = website.WebsiteIn.Count(x => x.Unique)
               let ratings = website.WebsiteRating.Select(x => x.Rating)
               let avgRate = ratings.Any() ? (int)ratings.Average() : 5
               let amountRedirects = website.WebsiteOut.Count(x => x.Unique)
               where
                   website.Enabled && website.Users.IsAdminVerified && website.Users.IsEmailVerified &&
                   (website.Users.BannedEndDate == null || DateTime.Now > website.Users.BannedEndDate)
               orderby website.Sponsored descending,
                   amountVotes descending,
                   amountRedirects descending,
                   avgRate descending,
                   website.DateAdded descending
               select website);

Database logic:

• Users -> (multiple) Website -> WebsiteIn & WebsiteRating & WebsiteIn & WebsiteOut

Answer 1

I found a massive query which only contains little logic.

Not at all, it contains lots of logic:

• 4 let statements that each cause aggregating subqueries.
• 4 orderby clauses that use the results of the let statements

For the SQL statement to be able to order by the subquery results, it should nest the SELECT statements. Without nesting, it needs to repeat the aggregate queries in the ORDER BY phrase. So believe it or not, this already is a somewhat optimized query shape.

why it selects so many columns

The end result is a list of complete Website objects, so all columns in the corresponding table should present in the result set. These columns ultimately originate from the inner-most query.

So there's not much in there that could be optimized easily, esp. not by a query generator.

If the query causes you trouble performance-wise, you could consider to

1. Return a projection, ie a subset of the returned columns (that is: not Website objects, but a type with less properties).
2. Return the results without ordering, in a type that contains the aggregate numbers, and do the ordering in memory. This will flatten the query shape and take some effort away from the query optimizer.