Python: Writing a progress bar to a file

Question

I have a python script that, as part of it, runs a perl script. I'm executing and capturing the stdout and stderr using the subprocess module; the pseudo code is:

import subprocess as sp
import sys
perl_script='/usr/.../foo.pl'
p = sp.Popen(perl_script,stdout=sp.PIPE,stderr=sp.PIPE)
fname='test.log'
shell_out=True
text = ''
with open(fname, 'w') as log:
    while True:
        data = p.stdout.read(1)
        if data == '' and p.poll() != None:
            break
        if data != '':
            if shell_out is True:
                sys.stdout.write(data)
                sys.stdout.flush()
            log.write(data)
    p.stdout.close()

For each iteration of the loop, data seems to be a single string character.

The issue I'm having is that this perl script displays a progress bar that it overwrites; something like:

[#####.....] 50%

and updates every 5%. As such, the log file I write to ends up having something along the lines of:

[#.........] 5% [##........] 10% [###.......] 15%

and so on.

Is there a method or way for me to write over the previous line in my log file, or is my only option to capture all of the text, try to modify it, then write it to a log file?

Edit: I think the simple nature of the example I gave might not tell the entire story. This perl script has multi progress bars (one for each task) and outputs additional information that I want to log. A more detailed example is:

NOTICE: one or more of things did stuff and you probably want to take note of it
writing the foo file . . .
    writing things . . .                             [####################] 100%
    writing stuff . . .                              [####################] 100%
    writing items . . .                              [####################] 100%
done
creating other files...
done

Keep in mind that the above output is an example that matches the characteristics of the stdout of this perl script, and that the real output (which I cannot post) is about 100 or so lines.

Finally, thanks for the formatting help :)

Edit #2: I've given up on trying to write the log file as stdout is being captured, and will instead store the stdout in a variable and write the log file after the script has finished. The only issue that remains is how to properly format the string so that the log file will be written correctly. The updated pseudo code is:

import subprocess as sp
import sys
tg_post='/usr/.../foo.pl'
p = sp.Popen(tg_post,stdout=sp.PIPE)
text = ''
fname='test.log'
shell_out=True
while True:
    data = p.stdout.read(1)
    if data == '' and p.poll() != None:
        break
    if data != '':
        if shell_out is True:
            sys.stdout.write(data)
            sys.stdout.flush()
        text += data        
p.stdout.close()
text = text.split('\r')
with open(fname, 'w') as log:
    log.write(text)

At the moment, this code results in a line for each percentage of each status bar. I suppose I could loop over each index in text (post-split) and remove an indices where there is a # and not a 100% , but I'm hoping there is something more robust. If I print text (post-split), I get what I want on the screen. I know it's possible to use print to write to a file, but I'd like to avoid that (I believe that method isn't used anymore).

Answer 1

You could define data outside of the while loop and write the last status to the file at the end. This way it will just write the last status update held by data to the file, which will avoid constantly overwriting the last status of the file.

import subprocess as sp
import sys
perl_script='/usr/.../foo.pl'
p = sp.Popen(perl_script,stdout=sp.PIPE,stderr=sp.PIPE)
fname='test.log'
shell_out=True
text = ''
data = ''

while True:
    data = p.stdout.read(1)
    if data == '' and p.poll() != None:
        break
    if data != '':
        if shell_out is True:
            sys.stdout.write(data)
            sys.stdout.flush()
    p.stdout.close()
with open(fname, 'w') as log:
    log.write(data)