Recursive python generators: why does the yield need to be iterated over?

Question

A good exercise to test one's understanding of recursion is to write a function that generates all permutations of a string:

def get_perms(to_go, so_far=''):
    if not to_go:
        return [so_far]
    else:
        ret = []
        for i in range(len(to_go)):
            ret += get_perms(to_go[:i] + to_go[i+1:], so_far + to_go[i])
    return ret 

This code is fine, but we can significantly improve the efficiency from a memory standpoint by using a generator:

def perms_generator(to_go, so_far=''):
    if not to_go:
        yield so_far
    else:
        for i in range(len(to_go)):
            for perm in perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i]):
                yield perm

(Note: the last for loop can also be replaced with yield from in python 3.3)

My question: why do we need to iterate over the results of each recursive call? I know that yield returns a generator, but from the statement yield so_far it would seem as though we're getting a string, and not something we would need to iterate over. Rather, it would seem as though we could replace

for perm in perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i]):
    yield perm

with

yield perms_generator(to_go[:i] + to_go[i+1:], so_far + to_go[i])

Thank you. Please let me know if the title is unclear. I have a feeling the content of this question is related to this SO question .

Answer 1

Remember, any function using yield does not return those values to the caller. Instead a generator object is returned and the code itself is instead paused until you iterate over the generator. Each time a yield is encountered, the code is paused again:

>>> def pausing_generator():
...     print 'Top of the generator'
...     yield 1
...     print 'In the middle'
...     yield 2
...     print 'At the end'
... 
>>> gen = pausing_generator()
>>> gen
<generator object pausing_generator at 0x1081e0d70>
>>> next(gen)
Top of the generator
1
>>> next(gen)
In the middle
2
>>> next(gen)
At the end
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

Calling the pausing_generator() function returned a generator object . Only iterating (using the next() function here) runs the actual code in the function, but pausing execution each time a yield is encountered.

Your perms_generator function returns a such a generator object, and a recursive call would still return a generator object. You could yield the whole generator object, but then you are producing a generator that produces a generator, etc. until you come to the inner-most generator.

You can visualise this with print statements:

>>> def countdown(i):
...     if not i:
...         return
...     yield i
...     recursive_result = countdown(i - 1)
...     print i, recursive_result
...     for recursive_elem in recursive_result:
...         yield recursive_elem
... 
>>> for i in countdown(5):
...     print i
... 
5
5 <generator object countdown at 0x1081e0e10>
4
4 <generator object countdown at 0x1081e0e60>
3
3 <generator object countdown at 0x1081e0eb0>
2
2 <generator object countdown at 0x1081e0f00>
1
1 <generator object countdown at 0x1081e0f50>

Here, the recursive calls returned a new generator object; if you wanted to have the elements produced by the generator your only choice is to loop over it and hand the elements down, not the generator object itself.

In Python 3, you can use yield from to delegate to a nested generator, including a recursive call:

def perms_generator(to_go, so_far=''):
    if not to_go:
        yield so_far
    else:
        for i, elem in enumerate(to_go):
            yield from perms_generator(to_go[:i] + to_go[i+1:], so_far + elem)

When encountering a yield from iteration continues into the recursive call instead of yielding the whole generator object.

Answer 2

The difference between return and yield is that the former just returns a value. The latter means "wrap the value in a generator and then return the generator."

So in all cases, the function perms_generator() returns a generator.

The expression yield perms_generator() again would wrap the result of perms_generator() in a generator, giving you a generator of a generator. That would mean the function would return different things; sometimes, it would be a simple generator and sometimes nested generators. That would be very confusing for the consumer of your code.