Python progress bar through logging module

Question

I have seen different solutions for a progress bar within Python, but the simple stdout solutions are not working for my project. I have multiple classes and use the "logging" module to output information to stdout. I have a function of which I want to show a progress bar on one line, flushing the buffer each time.

Example of the simple progress:

for i in range(100):
    time.sleep(1)
    sys.stdout.write("\r%d%%" %i)
    sys.stdout.flush()

When I try to write via stdout and then flush the buffer, either the buffer is not flushed or the progress doesn't go anywhere. I am hoping to avoid some sort of threading or complicated process to make this possible. Does someone have a preferred way of making this happen?

Answer 1

I couldn't find a good solution for this, so I wrote enlighten progress bar to handle it. Basically it changes the scroll area of the terminal so logging is done above the progress bar(s) rather than having to redraw the progress bar(s) every time you want to write to STDOUT. This lets you write to the terminal as much as you want without having to modify logging , print , etc.

import logging
import time
import enlighten

# Setup logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger()

# Setup progress bar
manager = enlighten.get_manager()
pbar = manager.counter(total=100, desc='Ticks', unit='ticks')

for i in range(1, 101):
    logger.info("Processing step %s" % i)
    time.sleep(.2)
    pbar.update()

Answer 2

I solved it like this:

import logging
import time
from tqdm import tqdm
import io

class TqdmToLogger(io.StringIO):
    """
        Output stream for TQDM which will output to logger module instead of
        the StdOut.
    """
    logger = None
    level = None
    buf = ''
    def __init__(self,logger,level=None):
        super(TqdmToLogger, self).__init__()
        self.logger = logger
        self.level = level or logging.INFO
    def write(self,buf):
        self.buf = buf.strip('\r\n\t ')
    def flush(self):
        self.logger.log(self.level, self.buf)

if __name__ == "__main__":
    logging.basicConfig(format='%(asctime)s [%(levelname)-8s] %(message)s')
    logger = logging.getLogger()
    logger.setLevel(logging.DEBUG)

    tqdm_out = TqdmToLogger(logger,level=logging.INFO)
    for x in tqdm(range(100),file=tqdm_out,mininterval=30,):
        time.sleep(.5)

Output

2016-12-19 15:35:06 [INFO    ] 16%|#####9                                | 768/4928 [07:04<40:50,  1.70it/s]
2016-12-19 15:36:07 [INFO    ] 18%|######6                               | 865/4928 [08:04<40:34,  1.67it/s]

Answer 3

You can use the tqdm progress bar with a custom handler through logging as described here :

import logging
import time
import colorlog
from tqdm import tqdm

class TqdmHandler(logging.StreamHandler):
    def __init__(self):
        logging.StreamHandler.__init__(self)

    def emit(self, record):
        msg = self.format(record)
        tqdm.write(msg)

if __name__ == "__main__":
    for x in tqdm(range(100)):
        logger = colorlog.getLogger("MYAPP")
        logger.setLevel(logging.DEBUG)
        handler = TqdmHandler()
        handler.setFormatter(colorlog.ColoredFormatter(
            '%(log_color)s%(name)s | %(asctime)s | %(levelname)s | %(message)s',
            datefmt='%Y-%d-%d %H:%M:%S',
            log_colors={
                'DEBUG': 'cyan',
                'INFO': 'white',
                'SUCCESS:': 'green',
                'WARNING': 'yellow',
                'ERROR': 'red',
                'CRITICAL': 'red,bg_white'},))

        logger.addHandler(handler)
        logger.debug("Inside subtask: "+str(x))
        time.sleep(.5)

Answer 4

If you know the progress bar is always going to be written to STDOUT, you should just be using print instead of a logger. See the documentation in the Python logging tutorial documentation

Answer 5

You might want to use progressbar2 which allows to keep logging while having the progressbar printed:

Minimal example:

import time
import progressbar

for i in progressbar.progressbar(range(100), redirect_stdout=True):
    print('Some text', i)
    time.sleep(0.1)

Will result in:

(progressbar2 is compatible with both Python2 and Python3)

Answer 6

Cleaning-up the code on these proposals, this is IMO the correct implementation, with no external dependencies other than tqdm (also posted here ):

import logging
from tqdm import tqdm


class TqdmLoggingHandler(logging.StreamHandler):
    """Avoid tqdm progress bar interruption by logger's output to console"""
    # see logging.StreamHandler.eval method:
    # https://github.com/python/cpython/blob/d2e2534751fd675c4d5d3adc208bf4fc984da7bf/Lib/logging/__init__.py#L1082-L1091
    # and tqdm.write method:
    # https://github.com/tqdm/tqdm/blob/f86104a1f30c38e6f80bfd8fb16d5fcde1e7749f/tqdm/std.py#L614-L620

    def emit(self, record):
        try:
            msg = self.format(record)
            tqdm.write(msg, end=self.terminator)
        except RecursionError:
            raise
        except Exception:
            self.handleError(record)

Testing:

import time

log = logging.getLogger(__name__)
log.setLevel(logging.INFO)
log.addHandler(TqdmLoggingHandler())
#   ^-- Assumes this will be the unique handler emitting messages to sys.stdout.
#       If other handlers output to sys.stdout (without tqdm.write),
#       progress bar will be interrupted by those outputs

for i in tqdm(range(20)):
    log.info(f"Looping {i}")
    time.sleep(0.1)