Python logging in multiprocessing: AttributeError: 'Logger' object has no attribute 'flush'

Question

Based on this code I created a python object that both prints output to the terminal and saves the output to a log file with the date and time appended to its name:

import sys
import time

class Logger(object):
    """
    Creates a class that will both print and log any
    output text. See https://stackoverflow.com/a/5916874
    for original source code. Modified to add date and
    time to end of file name.
    """
    def __init__(self, filename="Default"):
        self.terminal = sys.stdout
        self.filename = filename + ' ' + time.strftime('%Y-%m-%d-%H-%M-%S') + '.txt'
        self.log = open(self.filename, "a")

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)


sys.stdout = Logger('TestLog')

This works great, but when I try to use it with a script that uses the Pool multiprocessing function, I get the following error:

AttributeError: 'Logger' object has no attribute 'flush'

How can I modify my Logger object so that it will work with any script that runs in parallel?

Answer 1

If you're replacing sys.stdout , it must be with a file-like object , which means you have to implement flush . flush can be a no-op :

def flush(self):
    pass

Answer 2

@ecatmur's answer will only fix the missing flush, once this is added I receive:

AttributeError: 'Logger' object has no attribute 'fileno'

A comment in the post for the original code provides a modification which will account for any additional missing attributes, for completeness I will post the full working version of this code:

class Logger(object):
    def __init__(self, filename = "logfile.log"):
        self.terminal = sys.stdout
        self.log = open(filename, "a")

    def __getattr__(self, attr):
        return getattr(self.terminal, attr)

    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)

    def flush(self):
        pass

This runs for me without error in Python 2.7. Have not tested in 3.3+.

Answer 3

If the flush is not implemented, it won't work, find below a detailed answer with flush implemented.

import sys
import logging

class Logger(object):
    def __init__(self, filename):
        self.terminal = sys.stdout
        self.log = open(filename, "a")
    def __getattr__(self, attr):
        return getattr(self.terminal, attr)
    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)      
    def flush(self):
        self.terminal.flush()
        self.log.flush()

I was experiencing issues without the flushing the logs. Then, it can be used as follow:

sys.stdout = Logger("file.log")