Nested list comprehension with generators

Question

I have some strange behavior on python 3.7 with a nested list comprehension that involves a generator.

This works:

i = range(20)
n = [1, 2, 3]
result = [min(x + y for x in i) for y in n]

It does not work if i is a generator:

i = (p for p in range(20))
n = [1, 2, 3]
result = [min(x + y for x in i) for y in n]

This raises a ValueError: min() arg is an empty sequence

Now even if the generator i is wrapped with list it still creates the same error:

i = (p for p in range(20))
n = [1, 2, 3]
result = [min(x + y for x in list(i)) for y in n]

Is this a python bug or is it expected behavior? If it is expected behavior, can you explain why this does not work?

Answer 1

In both of your last examples, you try to iterate on the generator again after it got exhausted.

In your last example, list(i) is evaluated again for each value of y , so i will be exhausted after the first run.

You have to make a list of the values it yields once before, as in:

i = (p for p in range(20))
n = [1, 2, 3]
list_i = list(i)
result = [min(x + y for x in list_i) for y in n]

Answer 2

In i = range(20) the range(20) is a promise to generate a generator. While i = (p for p in range(20)) is already a generator.

Now write your list expression as:

for y in [1, 2, 3]:
    print(min(x + y for x in i))
## 1
## ...
## ValueError: min() arg is an empty sequence 

You get a 1 printed, but (the generator is exhausted in the first call) and then you get in the next round a ValueError: min() arg is an empty sequence because the generator i was already consumed in the first for-loop call for y as 1. While if i is defined as range(20) , everytime the for x in i is called, the generator is re-created again and again.

You can imitate what range(20) is doing by:

def gen():
    return (p for p in range(20))

for y in [1, 2, 3]:
    print(min(x + y for x in gen())) 
    # range() like gen() is a promise to generate the generator
## 1
## 2
## 3

Now the generator is created everytime anew.

But in fact, range is even cooler, if you do:

i = range(20)

for y in [1, 2, 3]:
    print(min(x + y for x in i))
## 1
## 2
## 3

The i inside the innerst generator is not a function call. But despite of that it creates - when evaluted - a new generator - at least when used as an iterable within a for loop.

This is actually implemented in Python using a class and by defining the __iter__() method. Which defines the behaviour in interators - here especiall a lazy behavior.

To imitate this behavior, we can generate a lazy generator ( lazy_gen ).

class lazy_gen:
    def __init__(self):
        pass

    def __iter__(self):    # everytime when used as an iterator
        return self.gen()  # recreate the generator # real lazy behavior

    def gen(self):
        return (p for p in range(20))

Which we can use like:

i = lazy_gen()

for y in [1, 2, 3]:
    print(min(x + y for x in i))
## 1
## 2
## 3

So this reflects even better the range() behavior.

Other languages (functional languages) like Lisp family languages (common-lisp, Racket, Scheme, Clojure), R , or Haskell have a better control over evaluation - thus over lazy evaluation and promises. But in Python, for such implementations and fine grained control, one has to take resort in OOP.

My range function and class

Finally, I figured out how the range function must have been realized roughly. (For fun, though I could have looked it up in the source code of Python I know - but sometimes reasoning is fun.)

class Myrange:
    def __init__(self, start, end, step):
        self.start = start
        self.end = end
        self.step = step

    def __iter__(self):
        return self.generate_range()

    def generate_range(self):
        x = self.start - self.step
        while x + self.step < self.end:
            x = x + self.step
            yield x

    def __repr__(self):
        return "myrange({}, {})".format(self.start, self.end)



def myrange(start=None, end=None, step=1):
    if start is None and end is None:
        raise "Please provide at least one number for the range-limits."
    elif start is not None and end is None:
        _start = 0
        _end = start
    elif start is not None and end is not None:
        _start = start
        _end = end
    else:
        _start = 0
        _end = end
    _step = step
    return Myrange(_start, _end, _step)

One can use it exactly like the range function.

i = myrange(20)

n = [1, 2, 3]
result = [min(x + y for x in i) for y in n]

result 
## [1, 2, 3]

i 
## myrange(0, 20)  # representation of a Myrange object.

myrange(20)
## myrange(0, 20)

list(myrange(3, 10))
## [3, 4, 5, 6, 7, 8, 9]

list(myrange(0, 10))
## [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

list(myrange(10))
## [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

list(myrange(0, 10, 2))
## [0, 2, 4, 6, 8]

list(myrange(3, 10, 2))
## [3, 5, 7, 9]

Answer 3

The generator is emptied after the first for loop for both for x in i or for x in list(i) , instead you need to convert the generator to a list, (which essentially iterates over the generator and empties it) beforehand and use that list

Note that this essentially defeats the purpose of a generator, since now this becomes the same as the first approach

In [14]: list(range(20)) ==  list(p for p in range(20))                                                                                                                             
Out[14]: True

Hence the updated code will be

#Create generator and convert to list
i = list(p for p in range(20))

n = [1, 2, 3]
#Use that list in the list comprehension
result = [min(x + y for x in i) for y in n]
print(result)

The output will be

[1, 2, 3]

Hence the better approach hence is to stick with the first approach itself, or you can have the generator inline, which, again is the same as the first approach with range

n = [1, 2, 3]
result = [min(x + y for x in (p for p in range(20))) for y in n]
print(result)
#[1, 2, 3]