Why does pdb give NameError for variables referenced in “if” clause of list comprehension?

Question

Why does this NameError happen?

(Blank lines before prompts inserted for readability.)

$ python3
Python 3.4.10 (default, Oct  4 2019, 19:39:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-23)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import pdb

>>> def blah():
...     foo = "ab"
...     pdb.set_trace()
... 

>>> blah()
--Return--
> <stdin>(3)blah()->None

(Pdb) [bar for bar in "ac" if bar in foo]
*** NameError: name 'foo' is not defined

All of the following work okay:

(Pdb) foo
'ab'

(Pdb) [bar for bar in foo]
['a', 'b']

(Pdb) [bar for bar in "ac" if bar in "ab"]
['a']

So it is specifically an issue for variables which are referenced in the if clause, other than the loop variable itself.

Same behaviour as above also seen in python 3.6.9 (default, Apr 18 2020, 01:56:04) .

But in python 2 (2.6.6 or 2.7.17), using the same commands as above, I get:

(Pdb) [bar for bar in "ac" if bar in foo]
['a']

Answer 1

List comprehensions in Python3 are evaluated with a separate scope, similar to functions. Only the first iterable ( for... in iterable ) is injected from the outside, all other names are bound as globals or closures. However, closures are resolved at compile time – they only work when lexically defined in the outer scope containing a name.

Compare this with defining an "inner" function, which shows the same behaviour:

-> pdb.set_trace()
(Pdb) foo
'ab'
(Pdb) def bar(): return foo
(Pdb) bar()
*** NameError: name 'foo' is not defined

In the PDB session, the execution scope is inside blah , but the lexical scope is outside. This is similar to defining an "inner" function/comprehension inside or outside its parent function:

# lexically scoped `foo` as closure
def blah():
    foo = 3
    def bar():  # !!! Define new scope inside blah
        print(foo)
    bar()  # execute bar inside blah

# `foo` not in scope
def bar():  # !!! Define new scope outside blah
    print(foo)

def blah():
    foo = 3
    bar()  # execute bar inside blah

---

6.2.4. Displays for lists, sets and dictionaries

[...]

However, aside from the iterable expression in the leftmost for clause, the comprehension is executed in a separate implicitly nested scope. This ensures that names assigned to in the target list don't “leak” into the enclosing scope.

4.2.2. Resolution of names

[...]

When a name is used in a code block, it is resolved using the nearest enclosing scope. The set of all such scopes visible to a code block is called the block's environment.