What does “<exception caught here>” line in twisted traceback mean?

Question

For example:

Unhandled error in Deferred:
Unhandled Error
Traceback (most recent call last):
  File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 388, in errback
    self._startRunCallbacks(fail)
  File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 455, in _startRunCallbacks
    self._runCallbacks()
  File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 542, in _runCallbacks
    current.result = callback(current.result, *args, **kw)
  File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 1076, in gotResult
    _inlineCallbacks(r, g, deferred)
--- <exception caught here> ---
  File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 1018, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/opt/zenoss/lib/python/twisted/python/failure.py", line 352, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
exceptions.TypeError: exceptions must be classes, or instances, not str

If it is caught there, why it still propagates two frames below? And how to find out from where it was thrown?

Answer 1

This isn't actually a traceback. This is how a twisted.python.failure.Failure "prints" itself (though to be sure things are ambiguous, the method for doing this is printTraceback ).

The reason the line --- <exception caught here> --- is sitting in the middle is that it is dividing information that relates to two different parts of the call stack related to the exception.

Above the marker the stack frames are describing what's above the exception handler that decided to create a Failure instance (from the exception and traceback state).

For example, consider this simple program:

from twisted.python.failure import Failure

def a():
    b()

def b():
    try:
        c()
    except:
        return Failure()

def c():
    1 / 0

f = a()
f.printTraceback()

If you run this then you'll see that above the marker line you get:

File "someprog.py", line 16, in <module>
  f = a()
File "someprog.py", line 5, in a
  return b()

If the call stack is a stack then you can see that these are the two frames that were pushed onto it before the frame that caught the exception and created the Failure .

And below the marker line you get:

File "someprog.py", line 9, in b
  c()
File "someprog.py", line 14, in c
  1 / 0

These are the two frames that were on the call stack and got popped off by the exception being raised.

Most often, only the frames below the marker are interesting. The frames above the marker are separated in this format because they're usually deep implementation details of some part of Twisted. Since Twisted is a cooperative multitasking system, the stack frames in the implementation often don't tell you much that's very useful about the context of the exception (because cooperative multitasking systems in Python can't keep a correspondence between the logical operation being processed and the Python frame objects that are doing the processing).

On top of this, understanding your traceback is made more complicated by the fact that the code involved uses inlineCallbacks . This plays further tricks with the call stack and usually ruins any tracebacks you get out.

File "/opt/zenoss/lib/python/twisted/internet/defer.py", line 1018, in _inlineCallbacks
  result = result.throwExceptionIntoGenerator(g)
File "/opt/zenoss/lib/python/twisted/python/failure.py", line 352, in throwExceptionIntoGenerator
  return g.throw(self.type, self.value, self.tb)
exceptions.TypeError: exceptions must be classes, or instances, not str

However, what I notice about this is that it is the attempt to raise an exception into a generator by the implementation of inlineCallbacks that is itself raising another exception.

This TypeError means that self.type is an instance of str . From this I would guess either that some of the code involved is mis-using Failure or that this application is using Twisted Spread (which mis-uses Failure ). The short answer I would prefer to give here is not to combine Twisted Spread and inlineCallbacks : they don't work nicely together.

Since I see Zenoss in the paths, I suspect that you did not write most of this code though, so that answer may not be of much help to you...

If you can, and if Twisted Spread is really involved here, you might try upgrading Twisted to see if this problem goes away. I think that this is a symptom of http://tm.tl/4520 which was fixed a couple years ago (fix first included in Twisted 11.1.0).

If Twisted Spread is not involved then you may need to track down the other code that is mis-using Failure and correct it.