Understanding Deferred Execution: Is a Linq Query Re-executed Everytime its collection of anonymous objects is referred to?

Question

I'm currently trying to write some code that will run a query on two separate databases, and will return the results to an anonymous object. Once I have the two collections of anonymous objects, I need to perform a comparison on the two collections. The comparison is that I need to retrieve all of the records that are in webOrders, but not in foamOrders. Currently, I'm making the comparison by use of Linq. My major problem is that both of the original queries return about 30,000 records, and as my code is now, it takes waay too long to complete. I'm new to using Linq, so I'm trying to understand if using Linq to compare the two collections of anonymous objects will actually cause the database queries to run over and over again - due to deferred execution. This may be an obvious answer, but I don't yet have a very firm understanding of how Linq and anonymous objects work with deferred execution. I'm hoping someone may be able to enlighten me. Below is the code that I have...

private DataTable GetData()
{
    using (var foam = Databases.Foam(false))
    {
        using (MySqlConnection web = new MySqlConnection(Databases.ConnectionStrings.Web(true)
        {
            var foamOrders = foam.DataTableEnumerable(@"
                    SELECT order_id
                    FROM   Orders
                    WHERE  order_id NOT LIKE 'R35%'
                    AND originpartner_code = 'VN000011'
                    AND orderDate > Getdate() - 7 ")
                .Select(o => new
                {
                    order = o[0].ToString().Trim()
                }).ToList();

            var webOrders = web.DataTableEnumerable(@"
                    SELECT ORDER_NUMBER FROM TRANSACTIONS AS T WHERE
                    (Str_to_date(T.ORDER_DATE, '%Y%m%d %k:%i:%s') >= DATE_SUB(Now(),  INTERVAL 7 DAY))
                    AND (STR_TO_DATE(T.ORDER_DATE, '%Y%m%d %k:%i:%s') <= DATE_SUB(NOW(),  INTERVAL 1 HOUR))")
                .Select(o => new
                {
                    order = o[0].ToString().Trim()
                }).ToList();
            return (from w in webOrders
                    where !(from f in foamOrders
                            select f.order).Contains(w.order)
                    select w
                ).ToDataTable();
        }
    }
}

Answer 1

Your linq ceases to be deferred when you do

ToDataTable();

At that point it is snapshotted as done and dusted forever.

Same is true with foamOrders and webOrders when you convert it

ToList();

You could do it as one query. I dont have mySQL to check it out on.

Answer 2

Regarding deferred execution:

Method .ToList() iterates over the IEnumerable retrieves all values and fill a new List<T> object with that values. So it's definitely not deferred execution at this point.

It's most likely the same with .ToDataTable();

PS But i'd recommend to :

1. Use custom types rather than anonymous types.
2. Do not use LINQ to compare objects because it's not really effective (linq is doing extra job)
3. You can create a custom MyComparer class (that might implement IComparer interface) and method like Compare<T1, T2> that compares two entities. Then you can create another method to compare two sets of entities for example T1[] CompareRange<T1,T2>(T1[] entities1, T2[] entities2) that reuse your Compare method in a loop and returns result of the operation

PS Some of other resource-intensive operations that may potentially lead to significant performance losses (in case if you need to perform thousands of operations) :

1. Usage of enumerator object ( foreach loop or some of LINQ methods)

Possible solution : Try to use for loop if it is possible.

2. Extensive use of anonymous methods (compiler requires significant time to compile the lambda expression / operator );

Possible solution : Store lambdas in delegates (like Func<T1, T2> )

Answer 3

In case it helps anyone in the future, my new code is pasted below. It runs much faster now. Thanks to everyone's help, I've learned that even though the deferred execution of my database queries was cut off and the results became static once I used .ToList(), using Linq to compare the resulting collections was very inefficient. I went with a for loop instead.

private DataTable GetData()
    {
        //Needed to have both connections open in order to preserve the scope of var foamOrders and var webOrders, which are both needed in order to perform the comparison.
        using (var foam = Databases.Foam(isDebug))
        {
            using (MySqlConnection web = new MySqlConnection(Databases.ConnectionStrings.Web(isDebug)))
            {
                var foamOrders = foam.DataTableEnumerable(@"
            SELECT foreignID
            FROM   Orders
            WHERE  order_id NOT LIKE 'R35%'
            AND originpartner_code = 'VN000011'
            AND orderDate > Getdate() - 7 ")
                                     .Select(o => new
                                     {
                                         order = o[0].ToString()
                                                     .Trim()
                                     }).ToList();


                var webOrders = web.DataTableEnumerable(@"
            SELECT ORDER_NUMBER FROM transactions AS T WHERE
                                                (Str_to_date(T.ORDER_DATE, '%Y%m%d %k:%i:%s') >= DATE_SUB(Now(),  INTERVAL 7 DAY))
                                                AND (STR_TO_DATE(T.ORDER_DATE, '%Y%m%d %k:%i:%s') <= DATE_SUB(NOW(),  INTERVAL 1 HOUR))
                                                 ", 300)
                            .Select(o => new
                            {
                                order = o[0].ToString()
                                            .Trim()

                            }).ToList();


                List<OrderNumber> on = new List<OrderNumber>();

                foreach (var w in webOrders)
                {
                    if (!foamOrders.Contains(w))
                    {
                        OrderNumber o = new OrderNumber();
                        o.orderNumber = w.order;
                        on.Add(o);
                    }
                }

                return on.ToDataTable();

            }
        }


    }


    public class OrderNumber
    {
        public string orderNumber { get; set; }
    }


 }