Python: Junk received when readline() a process called with Popen()

Question

I need to stream to a list the output of a file, I am getting the correct data but with a lot of junk at the end and don't understand what I am doing wrong.

The (what I thought was fairly simple) popen code is:

stream = []
p = Popen(["python", "-u", "test.py"], stdout=PIPE, bufsize=1)

while p.poll() is None:
    stream.append(p.stdout.readline())

print stream
print 'Returned: {0}'.format(p.returncode)

The output from calling this is:

The output from running it is:

['[INDEX 0] Current index position\n', '[INDEX 1] Current index position\n', '[INDEX 2] Current index position\n', '[INDEX 3] Current index position\n', '[INDEX 4] Current index position\n', 'exiting.
...\n', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '', '',
'', '', '', '', '', '', '', '', '', '', '', '', '', '']
Returned: -4

Test.py is:

index = 0

while True:
    print '[INDEX {0}] Current index position'.format(index)
    index += 1
    if index == 5: break

import sys
print 'exiting....'
sys.exit(-4)

Answer 1

The problem here is that your if-condition is incorrect:

while p.poll() is None:
    stream.append(p.stdout.readline())

This means "while the process is running, read a line". But that's not what you want. What you really want is "while there is more output, read a line". The code you wrote reads empty lines while the process is shutting down, so that's where all the empty strings come from.

What you should do instead is this:

for line in p.stdout:
    stream.append(line)

This correctly stops reading lines when there is no more output, and it is guaranteed to read all of the output. (Theoretically, your code could exit the loop before it has processed all of the output if the process produces output faster than you process it. You can emulate this by adding a time.sleep(0.1) to your loop.)