Retrieve data, strange behaviour

Question

I am using EF 6 to retrieve some data from SQL Server. I encountered this problem in the past, but I haven't asked anyone about it.

For the below code, just focus on the select.

Let's say in the database I have 2 rows:

1 2 3 4  5 
6 7 8 9 10

My code:

 var results = db.Table.Where(o => o.Version == 1)
                       .Select(o => new List<double?>{ o.M01, o.M02, o.M03, o.M04, o.M05});

 return results.ToList();

The code above will return a list of lists with the following:

previousDemand[0] = 1 2 3 4 5
previousDemand[1] = 10 9 8 7 6

The second list is reversed.

If I have more rows, it happens the same: first ok, second reversed, third ok, fourth reversed and so on.

If I change the code to this:

 var results = db.Table.Where(o => o.Version == 1).ToList();
 var x = results.Select(o => new List<double?>{ o.M01, o.M02, o.M03, o.M04, o.M05});

 return x.ToList();

everything will work fine here.

If seems there is a problem if I create lists within the EF query.

Am I missing something important?

UPDATE Adding some screenshots:

Answer 1

TL;DR

new List<double?> { o.M01, o.M02, o.M03, o.M04, o.M05 }.OrderBy(p => p).ToList()

That sugestion does only work on the sample you provided, but does not solve your problem, as pointed by @GertArnold on the comments.

What you could do instead is not query for a list, but instead return a new entity, with the columns you want. After materializing the result, you could turn it to a list.

var results = 
   db.Entities.Where(o => o.Version == 1)
     .Select(o => new { M01 = o.M01, M02 = o.M02, M03 = o.M03, M04 = o.M04, M05 = o.M05 });

return results.Select(o => new List<double?> { o.M01, o.M02, o.M03, o.M04, o.M05 });

Now the reasoning:

Your EF query generates the following (really messy) SQL statement:

SELECT 
    [Project6].[Id] AS [Id], 
    [Project6].[C2] AS [C1], 
    [Project6].[C1] AS [C2]
    FROM ( 
        SELECT 
            CASE 
                WHEN ([UnionAll4].[C1] = 0) THEN [Extent1].[M01] 
                WHEN ([UnionAll4].[C1] = 1) THEN [Extent1].[M02] 
                WHEN ([UnionAll4].[C1] = 2) THEN [Extent1].[M03] 
                WHEN ([UnionAll4].[C1] = 3) THEN [Extent1].[M04] 
                ELSE [Extent1].[M05] END AS [C1], 
            [Extent1].[Id] AS [Id], 
            1 AS [C2]
        FROM  [dbo].[Entities] AS [Extent1]
        CROSS JOIN  (
            SELECT 0 AS [C1]
            FROM  ( SELECT 1 AS X ) AS [SingleRowTable1]                
            UNION ALL               
                SELECT 1 AS [C1]
                FROM  ( SELECT 1 AS X ) AS [SingleRowTable2]                
            UNION ALL               
                SELECT 2 AS [C1]
                FROM ( SELECT 1 AS X ) AS [SingleRowTable3]             
            UNION ALL               
                SELECT 3 AS [C1]
                FROM  ( SELECT 1 AS X ) AS [SingleRowTable4]                
            UNION ALL               
                SELECT 4 AS [C1]
                FROM  ( SELECT 1 AS X ) AS [SingleRowTable5]
            ) AS [UnionAll4]
            WHERE 1 = [Extent1].[Version]
    )  AS [Project6]
ORDER BY [Project6].[Id] ASC, [Project6].[C2] ASC

As you can see on the ORDER BY clause, the query orders by [Project6].[C2], which is just a control flag that returns 1. This is because, as @marc_s specified in the comments, you didn't explicitly specify an order for your results.

If you would instead use somenting like:

var results = db.Entities.Where(o => o.Version == 1) .Select(o => new List { o.M01, o.M02, o.M03, o.M04, o.M05 }.OrderBy(p => p).ToList());

... then the generated SQL would include the correct order clause and you would get the exepcted result.

var results = 
   db.Entities.Where(o => o.Version == 1)
     .Select(o => new { M01 = o.M01, M02 = o.M02, M03 = o.M03, M04 = o.M04, M05 = o.M05 });

return results.Select(o => new List<double?> { o.M01, o.M02, o.M03, o.M04, o.M05 });

... then you wouldn't need to bother with the generated SQL order and would have control over the order your columns are returned on the output list.

Answer 2

This is a reproducible glitch in the way EF materializes a List<T> object from values originating from one database record. It turns out that the executed SQL statement returns the values in this unexpected alternating order. That means that EF generates a SQL query that is never guaranteed to return a specific order. But since SQL is a set-based language it never gives any guarantee as to the order of elements in a set. It would be possible to generate a query SQL that orders the elements correctly, but I think EF never anticipated your (not so common) scenario, or didn't think it worth the effort to support it.

Since it's part of EF's source code, there's no quick way to fix it and you'll have to settle with your work-around. You may want to file a bug report.

The root problem IMO is bad data normalization. If your table would be the parent of a table of M records --each representing one double value, maybe with a sequence field-- it would be much easier to obtain this list of lists.