How to save errors from Jupyter Notebook cells to a file?

Question

I am trying to save all output (stdout and all errors) of a cell to a file. To save stdout, I am using the following:

import sys
old_stdout = sys.stdout
sys.stdout = open('test.txt', 'w')
print("Hello World! ")

In this case, the output is not displayed and gets saved in the file as expected. To save errors, I used:

#Doesn't work
sys.stderr = open('error.txt','w')
print(a) #Should raise NameError

When I run this cell, I get the error in the notebook, and not in the file as expected:

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-5-de3efd936845> in <module>()
      1 #Doesn't work
----> 2 sys.stderr = open('error.txt','w')
      3 print("Test")
      4 print(a)

NameError: name 'sys' is not defined

I would like this saved in a file and not shown in the notebook. What is the correct code for this?

Answer 1

I think that the problem here is that IPython kernels spawned for the notebook use a ZMQInteractiveShell instance, which catches the errors before they make it to stderr, in order to send the error information to the various potential frontends (consoles, jupyter notebooks, etc). See ipykernel/ipkernel.py#L397-L413 for the code which catches exceptions, then InteactiveShell._showtraceback for the base implementation (print to sys.stderr ), and ZMQInteractiveShell._showtraceback for that used by notebook kernels (send stderr-channel messages over zmq to frontends).

If you're not bothered about getting exact stderr output, you could take advantage of IPython's existing error logging , which logs errors to a StreamHandler with the prefix "Exception in execute request:" . To use this, set the ipython log level, and alter the provided handler's stream:

import logging
import sys

my_stderr = sys.stderr = open('errors.txt', 'w')  # redirect stderr to file
get_ipython().log.handlers[0].stream = my_stderr  # log errors to new stderr
get_ipython().log.setLevel(logging.INFO)  # errors are logged at info level

Alternatively, to get your shell errors to print directly to a file without alteration, you can monkey-patch the _showtraceback method to print traceback to file as well as zmq message queues:

import sys
import types

# ensure we don't do this patch twice
if not hasattr(get_ipython(), '_showtraceback_orig'):
    my_stderr = sys.stderr = open('errors.txt', 'w')  # redirect stderr to file
    # monkeypatch!
    get_ipython()._showtraceback_orig = get_ipython()._showtraceback

    def _showtraceback(self, etype, evalue, stb):
        my_stderr.write(self.InteractiveTB.stb2text(stb) + '\n')
        my_stderr.flush()  # make sure we write *now*
        self._showtraceback_orig(etype, evalue, stb)

    get_ipython()._showtraceback = types.MethodType(_showtraceback, get_ipython())