How to inspect Generator type object?

Question

With below code (first case),

def f():
   mylist = range(3)
   for i in mylist:
      yield i*i

Without inspecting y , could you say, y=f() returns (x*x for x in range(3)) object of collections.abc.Generator type?

---

With the below code (second case),

def func():
    x = 1
    while 1:
        y = yield x
        x += y

What is that Generator type object returned when invoking y=func() ? How do you inspect y to see the code?

Answer 1

First case -- Simple Generator

The generator expression (x*x for x in range(3)) is roughly the same as the simple generator function you described. However, scoping for the genexp can be a little more complicated (which is why we usually recommend that you consume generator expressions immediately rather than passing them around).

Second Case -- Enhanced Generators

The code with the y = yield x is an example of an enhanced generator which is used to send data into a running generator, essentially creating a two-way communication channel between the running generator and the calling code.

Principal use cases for the send/receive logic are to implement coroutines and generator trampolines. See this trampoline example from David Beazley.

Enhanced generators are the key to Twisted Python's beautiful Inline Callbacks which implement coroutines .

How to Examine the Generator

For the variable y in y = func() , the only inspection technique is to examine the public API:

>>> y = func()
>>> dir(y)
['__class__', '__delattr__', '__doc__', '__format__',
 '__getattribute__', '__hash__', '__init__', '__iter__',
 '__name__', '__new__', '__reduce__', '__reduce_ex__',
 '__repr__', '__setattr__', '__sizeof__', '__str__',
 '__subclasshook__', 'close', 'gi_code', 'gi_frame',
 'gi_running', 'next', 'send', 'throw']

How to Inspect the Generator Function

For the generator function itself, you can use the dis module to inspect the code to see how it works:

>>> def func():
        x = 1
        while 1:
            y = yield x
            x += y

>>> import dis
>>> dis.dis(func)
  3           0 LOAD_CONST               1 (1)
              3 STORE_FAST               0 (x)

  4           6 SETUP_LOOP              21 (to 30)

  5     >>    9 LOAD_FAST                0 (x)
             12 YIELD_VALUE         
             13 STORE_FAST               1 (y)

  6          16 LOAD_FAST                0 (x)
             19 LOAD_FAST                1 (y)
             22 INPLACE_ADD         
             23 STORE_FAST               0 (x)
             26 JUMP_ABSOLUTE            9
             29 POP_BLOCK           
        >>   30 LOAD_CONST               0 (None)
             33 RETURN_VALUE 

Tracing the Code with a Debugger

You can use the pdb debugger to trace through the code step-by-step.

>>> import pdb
>>> y = func()
>>> pdb.runcall(next, y)
> /Users/raymond/Documents/tmp.py(2)func()
-> x = 1
(Pdb) s
> /Users/raymond/Documents/tmp.py(3)func()
-> while 1:
(Pdb) s
> /Users/raymond/Documents/tmp.py(4)func()
-> y = yield x
(Pdb) p locals()
{'x': 1}
(Pdb) s
> /Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/bdb.py(440)runcall()
-> self.quitting = 1
(Pdb) s
1
>>> pdb.runcall(y.send, 10)
> /Users/raymond/Documents/tmp.py(5)func()->1
-> x += y
(Pdb) s
> /Users/raymond/Documents/tmp.py(4)func()->1
-> y = yield x
(Pdb) locals()
{'__return__': 1, 'x': 11, 'y': 10}

Answer 2

The goal of a generator is to produce values on demand. In other words, if you need squares of integers, but you don't know in advance how many of them do you can use generators to generate values. It is achieved by using yield instead of return . After yield function does keep context. You can imagine yield as a "pause" and return as "stop". Another benefit of generators, even if you know what the size of your collection will be, is the fact that you don't load everything into the memory at once, but you consume just one element at a time.

Answer 3

In your second example you will get error for:

for i in func():
    print(i)

Because the y = yield x expression expects send() method call: see https://www.python.org/dev/peps/pep-0342/#specification-sending-values-into-generators

The "for" statement calls iter(foo()) function that returns the generator object. Then the "for" statement will call next(generator) while generator will not raise the StopIteration exception.

For your second example:

for i in func():
    print(i)

The "for" statement will get the generator object then call next(generator object) for him. For this call it will get the x value(1). At this time a generator object waits send(something) method call what will set the something value to y. If this method will not call, the next(generator object) call will send None to generator. In this case y will take a None value and program will raise Error on x += y

The proper use your second example is:

f = func()
# open generator object
next(f)
#1
f.send(2)
#3