python: logging output from external API to logger module

Question

I am python beginner. My python script logs output to a file (say example.log) using the basic python logging module. However, my python script also makes some 3rd party API calls (for example, parse_the_file) over which I don't have any control. I want to capture the output (usually on console) produced by the API into my example.log. The following example code works partially but the problem is that the contents get overwritten as soon as I start logging output of the API into my log file.

#!/usr/bin/env python

import logging
import sys
import os
from common_lib import *     # import additional modules

logging.basicConfig(filename='example.log', filemode='w', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

logging.debug('This is a log message.') # message goes to log file.

sys.stdout = open('example.log','a')
metadata_func=parse_the_file('/opt/metadata.txt') # output goes to log file but OVERWRITES the content
sys.stdout = sys.__stdout__
logging.debug('This is a second log message.') # message goes to log file.

I know there have been post suggesting to similar question on this site but I haven't a workaround/solution for this problem that will work in this scenario.

Answer 1

Try:

log_file = open('example.log', 'a')
logging.basicConfig(stream=log_file, level=logging.DEBUG)
logging.debug("Test")
sys.stdout = log_file
sys.stderr = log_file
stdout_fd = os.dup(1)
stderr_fd = os.dup(2)
os.dup2(log_file.fileno(), 1)
os.dup2(log_file.fileno(), 2)

os.system("echo foo")

os.dup2(stdout_fd, 1)
os.dup2(stderr_fd, 2)
sys.stdout = sys.__stdout__
sys.stderr = sys.__stderr__

However, this will not format it accordingly. If you want that, you can try something like http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-like-output-stream.html