How to stream data written to file to stdout

Question

I have shared library implemented in C which provides a function F . Every time I call F it logs its output in the file error.log .

Now I'm trying to capture the output produced by F from a python script (I'm using python 2.7.5 which I can't change for reasons out of my control).

I would like to stream the data written into error.log to a different file or stdout . I can't just open the file and parse it because it has more of stuff logged in, including the output of previous and later runs of F . I'm only interested in a specific execution which I can't be able to recognize from just the logging. That's why I'm trying to capture the output instead.

I tried opening error.log from python and then changing the file descriptor to make it point to stdout , but that doesn't seem to work (I tried the same with stdout and stderr and it did work).

What I'm doing is roughly

with open('error.log') as logfile:
    with redirect_output(logfile, sys.stdout):
        function_implemented_in_C()

where redirect_output is a context manager I implemented to do the redirection:

@contextmanager
def redirect_output(orig, dest):
    orig_fd = orig.fileno()
    with os.fdopen(os.dup(orig_fd)) as old_orig_fd:
        os.dup2(dest.fileno(), orig_fd)
    
    try:
        yield orig
    finally:
        # clean and restore fd's

I can't get this to work. Any ideas what I'm doing wrong?

UPDATE:

I reduced the problem to a simple script and it still doesn't seem to work. I guess it has something to do with the data being generated from a function in a shared lib (?) because if I do the same but redirecting the write calls from a file opened from python it works. This example works fine:

import sys
import os

def foo():
    f = open('dummy.txt', 'wb', buffering=0)
    os.dup2(sys.stdout.fileno(), f.fileno())
    f.write('some test data\n')
    f.close()

if __name__ == '__main__':
    foo()

But this doesn't

import sys
import os

def foo():
    f = open('error.log', 'wb', buffering=0)
    os.dup2(sys.stdout.fileno(), f.fileno())
    function_implemented_in_C()
    f.close()

if __name__ == '__main__':
    foo()

Answer 1

I'm answering my own question in case somebody stumble into this question with the same problem.

The issue here is that the C function producing the output in error.log launches a different process to perform the task. This makes not possible to easily redirect writes to the file to stdout , since the file descriptors are process specific.

So, if your function happens to produce the output using the same process, then the following should work

import sys
import os

def foo():
    f = open('dummy.txt', 'wb', buffering=0)
    os.dup2(sys.stdout.fileno(), f.fileno())
    f.write('some test data\n')
    f.close()

if __name__ == '__main__':
    foo()

If that's not the case then you can approach the problem by setting a loop reading from the file so it gets the newly written data. Something like (I haven't tried this code so it could not run):

from multiprocessing import Process, Pipe

def foo(filename, pipe_conn):
    with open(filename) as f:
        while True:
            line = f.readline()
            do_something(line)
            if pipe_conn.poll(0.01):
                break

def bar(pipe_conn):
    function_implemented_in_C()
    pipe_conn.send(['done'])
    pipe_conn.close()

if __name__ == '__main__':
    parent_conn, child_conn = Pipe()
    p = Process(target=bar, args=(child_conn,))
    p.start()
    foo(parent_conn)
    p.join()