Using lambdas with iteration values of for loop inside of that for loop in python

Question

I got major code smell from SonalLint for this code:

for subgroup_name in conf_data["groups_to_create"]:
        logging.info("Check if %s subgroup exists on created main group", subgroup_name)
        assert subgroup_exists_in_group(subgroup_name, main_group) is True
        variables_in_config = list(filter(lambda v: v["path"] == subgroup_name, conf_data["variables"]))
        variables_in_gitlab = glab.groups.get(
            list(filter(lambda b: b.name == subgroup_name, main_group.subgroups.list()))[0].id).variables.list()

It says "Functions and lambdas should not reference variables defined in enclosing loops" can someone explain what really happens under the hood and why we should not use lambda in side of for loop this way?

Answer 1

The issue with capturing variables of an enclosing loop, is that they have a somewhat surprising scope. Unlike, for example, most Lisp dialects, where there would be a new variable k for each turn through the loop, in Python there is a single variable which is reused throughout. Thus, if you do something like this (as juanpa.arravillaga mentions in a comment):

funcs = []
for k in range(5):
    funcs.append(lambda x: x * k)

and expect to get a list of functions that multiply by 0, 1, 2, etc, you are in for a nasty surprise. What you get is a list of functions which all multiply by 5, because that is the value of k once the loop has finished!

However, this is only a problem if the functions are allowed to "escape" out of the loop. Doing something like

for k in range(5):
    foo(list(filter(lambda x: x%k==0, somelist)))

is perfectly fine, because the value of k does not change during the lifetime of the function. Having said that, many — including me: — would probably argue that it would be more pythonic to write it with a list comprehension instead:

for k in range(5):
    foo([x for x in somelist if x%k==0])