How can I pass a child processes stdout with queue? Python

Question

I have a Logger class with the following:

class Logger():
    def __init__(self):
        self.terminal = sys.__stdout___
        self.log = open('logFile.log', 'w')
    def write(message):
        self.terminal.write(message)
        self.log.write(message)

a main with the following:

import Logger, sys
sys.stdout = Logger.Logger()
import File1
File1.aFunc()

in File1:

def execSubProc(target,queue):
    target(queue)

q = multiprocess.Queue()
proc = multiprocess.Process(target=execSubProc, args=(testFunc,q))
proc.start()
proc.join()

in File2:

def testFunc(queue)
#Some print statements
#I want to get these print statements in the queue for main process to read

Okay, so here is my structure. What I am trying to do is get the stdout from the child process running testFunc() and put it into the queue. When the program returns to main, i want to read from the queue and write those contents to the log file.

I can't just do sys.stdout = Logger.Logger() in file 2 again, because that will just overwrite mains log file. What can I do to capture all of child processes stdout and put in queue?

Answer 1

One solution could be to use the logging module.
A logger is disponible in multiprocessing.util and can be use to generate thread safe logs:

import time
import logging
import multiprocessing as mp
import multiprocessing.util as util
from sys import stdout as out


def runner(ids):
    log = util.get_logger()
    for i in range(10):
        time.sleep(.5)
        log.log(25, 'Process {} logging {}'.format(ids, i))

if __name__ == '__main__':
    # Setup the logger
    log = util.get_logger()
    log.getEffectiveLevel()
    # You can setup a file instead of stdout in the StreamHandler
    ch = logging.StreamHandler(out)
    ch.setLevel(25)
    ch.setFormatter(logging.Formatter('%(processName)s - %(message)s'))
    log.addHandler(ch)
    log.setLevel(25)

    p1 = mp.Process(target=runner, args=(1,), name="Process1")
    p2 = mp.Process(target=runner, args=(2,), name="Process2")

    p1.start()
    p2.start()

    time.sleep(2)
    log.log(25, 'Some main process logging meanwhile')

    p1.join()

The level >20 permits to avoid getting the logs from starting and stopping the processes.
The Handler can directly log in a file if the argument out is an open writable file.
This avoid having to process by yourself the log queue. There is also a lot of other high level functionalities in the logging module.

It is important to use the logger from the multiprocessing module to get the thread safe logs.