How to get logging messages

Question

import serial
import string
from time import sleep
from subprocess import call, check_call, CalledProcessError
import logging

import random
import signal
import os

LOG_FILENAME = "/tmp/dfu.log"
LOG_FD = open(LOG_FILENAME, "w")

logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG)

dfu_image_list = ['/tmp/sample.dfu']

while True:
    try:
        for test_file in dfu_image_list:
            logging.info("\n==================\nbegin dfu download test")
            check_call('sudo dfu-util -vD /tmp/sample.dfu', shell=True,  
                        stdout=LOG_FD, stderr=LOG_FD)
            logging.info("Download completed successfully!")
            sleep(5)   

    except CalledProcessError as e:
        msg = "dfu-util failed with return code :%s \n\nMessage:%s" % 
               (e.returncode, e.message)
        logging.warning(msg)
        logging.warning("USB device likely needs time to re-enumerate, 
                         waiting 10 seconds before restarting")
        sleep(10)

    except OSError:
        logging.error("dfu-util executable not found!")
        exit(1)

Execution of above python script provides logs into /tmp/dfu.log. However logs into log file are from the function check_call. Expected behavior is main threads logs like

logging.info("\n==================\nbegin dfu download test")
logs of function check_call
logging.info("Download completed successfully!").

However only logs of function check_call gets reflected and main threads logs like

begin dfu download test
Download completed successfully!

are not getting reflected into log file.

Answer 1

Remember that the logging module does some buffering, which means writing a log using log.something() doesn't necessarily means that this log will be written to the file.

Also, you're opening a file twice, and writing to it from different places. Usually that's a bad idea, even if Linux is preemptive and you flush the log and could possible work it still a bad idea.

What about you communicate() with the process instead of check_call() and then you log the stdout or stderr as you wish. For example:

for image in dfu_image_list:
    logging.info('=' * 10)
    logging.info('Begin dfu download using {}'.format(image))

    process = Popen(['sudo', 'dfu-util', '-vD', image], stdout=PIPE, stderr=PIPE)
    stdout, stderr = process.communicate()

    logging.info(stdout)
    if stderr:
        logging.warning(stderr)

    logging.info('Download completed successfully!')

By the way, your loop logic is flawed. As any error will restart the loop for dfu_image_list images.

I think this is more what you want to do:

from sys import exit
from time import sleep
from subprocess import Popen, PIPE, CalledProcessError
import logging


LOG_FILENAME = "/tmp/dfu.log"
ATTEMPTS = 3

logging.basicConfig(filename=LOG_FILENAME, level=logging.DEBUG)


def download(where):

    logging.info('=' * 10)
    logging.info('Begin dfu download to {}'.format(where))

    for attempt in range(ATTEMPTS):
        logging.info('Attempt #{}...'.format(attempt + 1))

        try:
            process = Popen(
                ['sudo', 'dfu-util', '-vD', where],
                stdout=PIPE, stderr=PIPE
            )
            stdout, stderr = process.communicate()

            logging.info(stdout)
            if stderr:
                logging.warning(stderr)

            logging.info('Download completed successfully!')
            return True

        except CalledProcessError as e:
            logging.warning(
                'dfu-util failed with return code {}'.format(e.returncode)
            )
            logging.warning(
                'Message:\n{}'.format(e.message)
            )
            logging.warning(
                'USB device likely needs time to re-enumerate, '
                'waiting 10 seconds before restarting...'
            )
            sleep(10)
            continue

        except OSError:
            logging.critical('dfu-util executable not found!')

    return False


if __name__ == '__main__':
    if not download('/tmp/sample.dfu'):
        exit(1)