Tracebacks in python 3.5

Question

To handle the errors in Python 3.5 gracefully, I am using the following code:

    except:
        exc_type, exc_value, exc_traceback = sys.exc_info()
        # print("Unable to write feed_item in database")
        # print(feeditem_insert_data)
        error_insert_data = (
            prog_name, 'Unable to populate item for (feed id,ref_feed_id) ('
                       + str(feed_id) +',' + str(ref_feed_id) + ') in feed_item table'
            , str(exc_type), str(exc_value), repr(traceback.format_tb(exc_traceback)), datetime.datetime.now()
        )
        cur_error_log.execute(error_insert_stmt,error_insert_data)
        continue

After adding this code at many places in my program, I am observing weird behavior as in program running long behaving on same data differently than program running short. I cant conclusively say this behavior showed up after adding the code above.

But I remember reading somewhere that if I use traceback than I have to deallocate memory manually.

Please advice if the module I have above is technically correct or something needs to be done to avoid issues.

Answer 1

The main change in Python 3 regarding exceptions is that it's no longer necessary to use sys.exc_info in order to get the traceback. It's now available as the __traceback__ attribute on the exception object (this is possible since all exceptions in Python 3 must inherit from the base Exception class, while older versions of Python 2 allowed anything to be treated as an exception).

So, you could replace the first two lines of your except code with except Exception as e: and then use type(e) , e , and e.__traceback__ in the error reporting code, rather than the values you were getting from sys.exc_info .

Having the traceback as an attribute on the exception can have some adverse consequences though on memory usage. Specifically, it creates a reference cycle between the current frame of execution and the exception. The frame references the exception (if you've bound it to a local variable), the exception references the traceback through its attribute and the traceback references the frame. The cycle means that the memory can't be reclaimed immediately by the reference counting system when it goes out of scope. Instead it has to rely upon the cyclical garbage collector, which might be slow to reclaim the memory, or even never get around to it (you can turn it off completely).

To reduce the impact of this issue, Python will automatically delete the exception variable created by a except ExceptionType as e statement at the end of its block. If you create references to the exception under some other name, you may need to clean them up yourself to avoid the memory sticking around longer than you need it.

So, if you really are having exception related memory issues, try adding del exc_value, exc_traceback at the end of the try block, just before continue . Or alternatively, switch to the except Exception as e syntax I described above, and e (or whatever name you use) will be removed from the local namespace for you automatically.