Generator callbacks on Twisted deferred

Question

I'm trying to jump around in the callback chain for a Twisted deferred, using callbacks that return generators. Consider the following snippet:

from twisted.internet import defer

def callback_one(result):
    print('Callback one: got "{}", will raise ZeroDivisionError'.format(result))
    raise ZeroDivisionError
    # yield

def errback_two(failure):
    print('Errback two: handled "{}", recovering'.format(failure.type))
    return 'recovered'

def callback_three(result):
    print('Callback three: got "{}"'.format(result))
    return 'Final result'

if __name__ == '__main__':
    d = defer.Deferred()
    d.addCallback(callback_one)
    d.addErrback(errback_two)
    d.addCallback(callback_three)
    d.callback('First result')

The output for that is

Callback one: got "First result", will raise ZeroDivisionError
Errback two: handled "<class 'ZeroDivisionError'>", recovering
Callback three: got "recovered"

However, if the yield is uncommented from callback_one I get only

Callback three: got "<generator object callback_one at 0x104603af0>"

If I understand correctly, what's happening is that the first callback returns a generator which is not evaluated until it's too late, the exception is not caught and therefore the errback is not invoked.

To sum up, the question is: if a callback returns a generator, how can I raise an exception from it in a way that is caught by the deferred object so the errback chain is triggered?

I'm a Twisted beginner so perhaps what I'm trying to do is a bad practice or even impossible/really hard to achieve, please let me know if that's the case. Thanks in advance!

Answer 1

If I understand correctly, to get the results such as the one you get from commenting out yield , you need to invoke the generator. To do that you could create another function that just executes the generator and make a minor change to your main function:

def exec_gen(gen):
    """
    Execute the generator
    """
    for x in gen:
        print(x)
# ...

if __name__ == '__main__':
    d = defer.maybeDeferred(exec_gen, callback_one('First result'))
    d.addErrback(errback_two)
    d.addCallback(callback_three)