How to show log statements of extension module in IPython Notebook

Question

I am working on a module which is hybrid C++ and Python, connected via Cython. The C++ code includes simple logging functionality which prints log statement to the console. When using the module via the Python or IPython command line, log statements are printed correctly. However, the module is meant to be used primarily from the IPython HTML Notebook. In the notebook, log statements do not appear. How can I fix this?

Answer 1

You can do so by using the ctypes library and hooking the libc stdout file descriptor: http://eli.thegreenplace.net/2015/redirecting-all-kinds-of-stdout-in-python/

I have tested this with cython and it works fine.

Reproducing the python3 code here for posterity:

from contextlib import contextmanager
import ctypes
import io
import os, sys
import tempfile

libc = ctypes.CDLL(None)
c_stdout = ctypes.c_void_p.in_dll(libc, 'stdout')

@contextmanager
def stdout_redirector(stream):
    # The original fd stdout points to. Usually 1 on POSIX systems.
    original_stdout_fd = sys.stdout.fileno()

    def _redirect_stdout(to_fd):
        """Redirect stdout to the given file descriptor."""
        # Flush the C-level buffer stdout
        libc.fflush(c_stdout)
        # Flush and close sys.stdout - also closes the file descriptor (fd)
        sys.stdout.close()
        # Make original_stdout_fd point to the same file as to_fd
        os.dup2(to_fd, original_stdout_fd)
        # Create a new sys.stdout that points to the redirected fd
        sys.stdout = io.TextIOWrapper(os.fdopen(original_stdout_fd, 'wb'))

    # Save a copy of the original stdout fd in saved_stdout_fd
    saved_stdout_fd = os.dup(original_stdout_fd)
    try:
        # Create a temporary file and redirect stdout to it
        tfile = tempfile.TemporaryFile(mode='w+b')
        _redirect_stdout(tfile.fileno())
        # Yield to caller, then redirect stdout back to the saved fd
        yield
        _redirect_stdout(saved_stdout_fd)
        # Copy contents of temporary file to the given stream
        tfile.flush()
        tfile.seek(0, io.SEEK_SET)
        stream.write(tfile.read())
    finally:
        tfile.close()
        os.close(saved_stdout_fd)

and in your code:

f = io.BytesIO()

with stdout_redirector(f):
    print('foobar')
    print(12)
    libc.puts(b'this comes from C')
    os.system('echo and this is from echo')
print('Got stdout: "{0}"'.format(f.getvalue().decode('utf-8')))

Answer 2

C/C++ printing to stdout cannot be redirected, only python one can. The only way would be to have your c/c++ library to accept a parameter to redirect its logging to a special file.