How to log errors/exceptions in a log file using logging module?

Question

I'm trying to debug a python script using logger. It opens another python script and checks whether it's running or not. If not then it restarts that script.

My script:

while True:
    print("Restart")
    logger.info("Restart")

    try:
        p = subprocess.Popen(["python", OPEN_FILE]).wait()
    except:
        logger.exception("Error opening script")
    print("Exit")
    logger.error("Exit")
    time.sleep(10)
    if p != 0:
        continue
    else:
        break

If file is not found then it prints error on terminal:

pi@raspberrypi:~/Desktop/MODBUS_TCP $ sudo python py_restart_script.py
Restart
python: can't open
file'/home/pi/Desktop/MODBUS_TCP/API_Modbus_TCP_Server3.py': [Errno 2] No
such file or directory
Exit

But that error is not in the log file:

2018-11-15 22:30:16,269 - INFO - Restart
2018-11-15 22:30:16,325 - ERROR - Exit

How to log same error showing in terminal to the log file?

Answer 1

What's printed in the terminal is process stderr output. You can get it in your python script and print it in the log file

>>> import subprocess
>>> p = subprocess.Popen(["python", "foobar"], stderr=subprocess.PIPE)
>>> ret = p.wait()
>>> ret # this holds the `subprocess` return code. ret != 0 indicates an error
2
>>> (_, stderr) = p.communicate()
>>> stderr
"python: can't open file 'foobar': [Errno 2] No such file or directory\n"
>>>