C# under the hood code generation for lambdas within loops

Question

This question is based on the answer given by one of my favourite posters Mehrdad Afshari in this question about closure.

I am having a hard time understand why C# generates the code the way it does......

Here is the code in question

    static void Main(string[] args)
    {

        List<string> list = new List<string> { "hello world",  "TED", "goodbye world" };

        IEnumerable<string> filteredList1 = list;

        IEnumerable<string> filteredList2 = list;

        var keywords = new[] { "hello", "world" };

        foreach (var keyword in keywords)
        {
            //Will execute the following 
            //filteredList1 = filteredList1.Where(item => item.Contains("hello")).Where(item => item.Contains("world"));;
            string value = keyword;
            filteredList1 = filteredList1.Where(item => item.Contains(value));

            //Will execute the following 
            //filteredList2 = filteredList2.Where(item => item.Contains("world"))
            filteredList2 = filteredList2.Where(item => item.Contains(keyword));
        }

        Console.WriteLine("===================================================");
        Console.WriteLine("LIST 1");
        foreach (var s in filteredList1)  // closure is called here
            Console.WriteLine(s);
        Console.WriteLine("===================================================");

        Console.WriteLine("LIST 2");
        foreach (var s in filteredList2)  // closure is called here
            Console.WriteLine(s);
        Console.WriteLine("===================================================");
    }
}

Gives the following output

===============
LIST 1
hello world
===============
LIST 2
hello world
goodbye world
===============

My problem is that I don't understand why filteredList2 list not generating the same code as filteredList1. It seems more sensible to that with each iteration of the foreach (var keyword in keywords) should just append another .Where(item => item.Contains(keyword)) and pass in a copy keyword current value.

Why does it not do this?

EDIT : OK maybe I was not clear. I understand When and how the closure generated, however I don't understand WHY it is done like this. Surely it makes sense hat if a compiler detects a loop variable is being used then why can it not generate a temp variable and ultimately end up in the same situation as filteredList1. Am I missing something here? May be there is some scenario where you would want to pass the same context to a lambda multiple times, but even then it always makes sense for the compiler to use local variable to store the value of loop variable when it is used as a context for a lambda.

To quote Jon Skeets definition of closure "To put it very simply, closures allow you to encapsulate some behaviour, pass it around like any other object, and still have access to the context in which they were first declared."

Surely you guys can see that c# closure over loop variable loses the context of the loop variable in which it was first set.

PS Please excuse the waffling I have a severe flu and trying to be concise is very difficult :-)

Answer 1

You have to remember that clousures close over variables, not values... So on each iteration, for the first filter, you're capturing the keyword value in the var 'value', which is what you're expecting; but for the second filter you are capturing the iteration variable 'keyword', so by the time it gets executed, all the filters have the same value for keyword (the last keyword in the iteration) 'world' and it's correctly showing the two entries that contain 'world'

Check out the following question for explanation

Why is it bad to use an iteration variable in a lambda expression

Answer 2

I think the key is understanding two things:

1. When and how is the closure generated? In this case the first closure is over a temporary variable called value , the second one over the loop variable
2. When is the lambda containing the closure executed? Here both lambdas are executed at the final foreach loops. Up until now they only had been assembled, but not executed. Now that they are executed each of them inspects the value of the closure, the first was using a temporary variable which is different for each step of the foreach loop (since it's a new variable each time). The second is using whatever the last value of the foreach loop variable was when it was built.