在 Python 中对子进程 PIPE 进行非阻塞读取，一次一个字节

Question

I have implemented a variant on the code in this question:

A non-blocking read on a subprocess.PIPE in Python

To try and read the output in real time from this dummy program test.py :

import time, sys

print "Hello there"
for i in range(100):
    time.sleep(0.1)
    sys.stdout.write("\r%d"%i)
    sys.stdout.flush()
print
print "Go now or I shall taunt you once again!"

The variation on the other question is that the calling program must read character by character, not line by line, as the dummy program test.py outputs progress indication all on one line by use of \\r . So here it is:

import sys,time
from subprocess import PIPE, Popen
from threading  import Thread

try:
    from Queue import Queue, Empty
except ImportError:
    from queue import Queue, Empty  # Python 3.x

ON_POSIX = 'posix' in sys.builtin_module_names

def enqueue_output(out, queue):
    while True:
        buffersize = 1
        data = out.read(buffersize)
        if not data:
            break
        queue.put(data)
    out.close()

p = Popen(sys.executable + " test.py", stdout=PIPE, bufsize=1, close_fds=ON_POSIX)
q = Queue()
t = Thread(target=enqueue_output, args=(p.stdout, q))
t.daemon = True # Thread dies with the program
t.start()

while True:
    p.poll()
    if p.returncode:
        break
    # Read line without blocking
    try:
        char = q.get_nowait()
        time.sleep(0.1)
    except Empty:
        pass
    else: # Got line
        sys.stdout.write(char)
        sys.stdout.flush()

print "left loop"
sys.exit(0)

Two problems with this

• It never exits - p.returncode never returns a value and the loop is not left. How can I fix it?
• It's really slow! Is there a way to make it more efficient without increasing buffersize ?

Answer 1

As @Markku K. pointed out, you should use bufsize=0 to read one byte at a time.

Your code doesn't require a non-blocking read. You can simplify it:

import sys
from functools import partial
from subprocess import Popen, PIPE

p = Popen([sys.executable, "test.py"], stdout=PIPE, bufsize=0)
for b in iter(partial(p.stdout.read, 1), b""):
    print b # it should print as soon as `sys.stdout.flush()` is called
            # in the test.py
p.stdout.close()
p.wait()

Note: reading 1 byte at a time is very inefficient.

Also, in general, there could be a block-buffering issue that sometimes can be solved using pexpect , pty modules or unbuffer , stdbuf , script command-line utilities .

For Python processes you could use -u flag to force unbuffering (binary layer) of stdin, stdout, stderr streams.