Output progress (percentage) to a file in python

Question

I need a way to estimate how long my program will take to complete. It is currently spending most its time in a nested for loop. I wish to see how many iterations have occured inside this for loop to get an idea of how long it will take to get a processed output file. Is there a way to output the % done to a file in python?

Here is my current approach, but when I try to read the progress file, it is always empty. I thought by closing the file, I would be able to see what has been written.

 48     #Estimate finishing time
 49     progTotal = len(f1)
 50     prog = 0
 51
 52     #Now to calculate 1st order Walsh Coefficients by subtracting away 2nd order coefficients
 53     for g in f1:
 54         progFile = open("progress",'w')
 55         sumOfOtherWalsh = 0
 56         for gPair in w2:
 57             if g == gPair[0] or g == gPair[1]:
 58                 sumOfOtherWalsh += w2.get(gPair, 0) #Add the walsh value into sum, if it is there
 59         f1[g] -= sumOfOtherWalsh #subtract all the Walsh values as well as the value in f1
 60         progFile.write(str(float(prog)/progTotal)+"\n")
 61         prog += 1
 62         progFile.close()

Surprisingly couldn't find information on this here, so I assume I am doing it wrong, but thanks for the help anyways.

Answer 1

As for displaying the progress of your computations to stdout you could use the progressbar module.

Example:

import time
import progressbar

maxval = 100
pbar = progressbar.ProgressBar(maxval=maxval)
pbar.start()
for i in range(maxval):
    # do something
    time.sleep(0.05)
    pbar.update(i+1)
pbar.finish()

Output:

$ python test.py
52% |#####################################                                    |

As for getting the output to file, it is not required that you close the file to get to see the output, but to flush the output buffers. (In fact it will be inefficient and probably counterproductive to open and possibly truncate the file repeatedly in the body of your loop.)

Have a look at the file object's flush and the os module's fsync methods for flushing the file content to disk. However, this should usually not be required. When you write enough data to the file, the buffers will be flushed as fast as you can print it to the terminal. (So that you can usually tail -f progess_file the output without a problem.)

If it's logging what you want to achieve, have a look at the logging module . You can use it to get a nice extendable logging system. You can select whether or not to do the logging to file or to stdout at a single point in the code. You can attach other modules' logging activities to the same log by just "getting" the main logger there by name:

logger = logging.getLogger("__main__")

(Alternatively, you can very well set up another logger using a different name there.)

Example:

import time
import logging

# set up the logging
def setup_logger(logfilename = None):
    logger = logging.getLogger(__name__) # probably __name__ == "__main__" 
    logger.setLevel(logging.DEBUG)
    formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')
    if logfilename:
        # write to logfile
        handler = logging.FileHandler(logfilename)
    else:
        # write to stdout
        handler = logging.StreamHandler()
    handler.setFormatter(formatter)
    logger.addHandler(handler)
    return logger
logger = setup_logger("test.log")

# example use of the logger
maxval = 100
for i in range(1,maxval+1):
    # do something
    time.sleep(0.05)
    if (i)%(maxval/10) == 0:
        logger.debug("progress:% 4d %% done", i)

Output:

2013-05-15 21:03:23,313 - DEBUG - progress:  10 % done
2013-05-15 21:03:23,822 - DEBUG - progress:  20 % done
2013-05-15 21:03:24,323 - DEBUG - progress:  30 % done
2013-05-15 21:03:24,825 - DEBUG - progress:  40 % done
2013-05-15 21:03:25,326 - DEBUG - progress:  50 % done
2013-05-15 21:03:25,827 - DEBUG - progress:  60 % done
2013-05-15 21:03:26,328 - DEBUG - progress:  70 % done
2013-05-15 21:03:26,829 - DEBUG - progress:  80 % done
2013-05-15 21:03:27,330 - DEBUG - progress:  90 % done
2013-05-15 21:03:27,831 - DEBUG - progress: 100 % done