composed filter() in python doesn't work as expected

Question

As far as i understand, the following code should output [['b']] . Instead, it outputs [['a', 'exclude'], ['b']] .

Is it a bug in python, or do I misunderstand something?

lists_to_filter = [
    ['a', 'exclude'],
    ['b']
]
# notice that when 'exclude' is the last element, the code returns the expected result
for exclude_label in ['exclude', 'something']:
    lists_to_filter = (labels_list for labels_list in lists_to_filter if exclude_label not in labels_list)
    # notice that changing the line above to the commented line below (i.e. expanding the generator to a list) 
    # will make the code output the expected result, 
    # i.e. the issue is only when using filter on another filter, and not on a list
    # lists_to_filter = [labels_list for labels_list in lists_to_filter if exclude_label not in labels_list]
lists_to_filter = list(lists_to_filter)
print(lists_to_filter)

Answer 1

It happens because lists_of_filter is only iterated outside the loop. Outside the loop you have exclude_label == 'something' , that is why you get unexpected results. To check it you can put a line exclude_label = 'exclude' :

lists_to_filter = [
    ['a', 'exclude'],
    ['b']
]

for exclude_label in ['exclude', 'something']:
    lists_to_filter = (labels_list for labels_list in lists_to_filter if exclude_label not in labels_list)

exclude_label = 'exclude'
lists_to_filter = list(lists_to_filter)
print(lists_to_filter)  # [['b']]

The doc for generator expressions says that " Subsequent for clauses and any filter condition in the leftmost for clause cannot be evaluated in the enclosing scope as they may depend on the values obtained from the leftmost iterable. ". In your case the filter condition if exclude_label ... depends on a value obtained from for exclude_label in ... loop.

Answer 2

because you multiple assign lists_to_filter in loop, so it will only return the last result. that is not include 'something' , the last element in ['exclude', 'something']

you can use all to achieve your goal:

lists_to_filter = [labels_list for labels_list in lists_to_filter if all(exclude_label not in labels_list for exclude_label in ['exclude', 'something'])]

or expand all logic:

    result = []

    for labels_list in lists_to_filter:
        exclude = False
        for exclude_label in ['exclude', 'something']:
            if exclude_label in labels_list:
                exclude = True
        if not exclude:
            result.append(labels_list)

    print(result)

or you can use filter according to your title:

lists_to_filter = list(filter(lambda x: all(exclude_label not in x for exclude_label in exclude_labels), lists_to_filter))

output:

[['b']]

Hope that helps you, and comment if you have further questions. : )