How do I get data from a subprocess PIPE while the subprocess is running in Python?

Question

I've got a program on Windows that calls a bunch of subprocesses, and displays the results in a GUI. I'm using PyQt for the GUI, and the subprocess module to run the programs.

I've got the following WorkerThread , that spawns a subthread for each shell command devoted to reading the process stdout and printing the results (later I'll wire it up to the GUI).

This all works. Except proc.stdout.read(1) never returns until after the subprocess has completed. This is a big problem, since some of these subprocesses can take 15-20 minutes to run, and I need to display results as they're running.

What do I need to do to get the pipe working while the subprocess is running?

class WorkerThread(QtCore.QThread):
    def run(self):
        def sh(cmd, cwd = None):
            proc = subprocess.Popen(cmd,
                                    shell = True,
                                    stdout = subprocess.PIPE,
                                    stderr = subprocess.STDOUT,
                                    stdin = subprocess.PIPE,
                                    cwd = cwd,
                                    env = os.environ)

            proc.stdin.close()

            class ReadStdOutThread(QtCore.QThread):
                def run(_self):
                    s = ''
                    while True:
                        if self.request_exit: return

                        b = proc.stdout.read(1)
                        if b == '\n':
                            print s
                            s = ''
                            continue

                        if b:
                            s += b
                            continue

                        if s: print s
                        return

            thread = ReadStdOutThread()
            thread.start()

            retcode = proc.wait()
            if retcode:
                raise subprocess.CalledProcessError(retcode, cmd)

            return 0

FWIW: I rewrote the whole thing using QProcess , and I see the exact same problem. The stdout receives no data, until the underlying process has returned. Then I get everything all at once.

Answer 1

If you know how long will be the the lines of command's output you can poll on the stdout PIPE of the process.

An example of what I mean:

import select
import subprocess
import threading
import os

# Some time consuming command.
command = 'while [ 1 ]; do sleep 1; echo "Testing"; done'

# A worker thread, not as complex as yours, just to show my point.
class Worker(threading.Thread):

    def __init__(self):
        super(Worker, self).__init__()
        self.proc = subprocess.Popen(
            command, shell=True, 
            stdout=subprocess.PIPE, 
            stdin=subprocess.PIPE, stderr=subprocess.STDOUT
        )

    def run(self):
        self.proc.communicate()


    def get_proc(self):
        # The proc is needed for ask him for his
        # output file descriptor later.
        return self.proc


if __name__ == '__main__':
    w = Worker()
    w.start()

    proc = w.get_proc()

    pollin = select.poll()

    pollin.register(proc.stdout, select.POLLIN)


    while ( 1 ):
        events = pollin.poll()
        for fd, event in events:
             if event == select.POLLIN:
                 # This is the main issue of my idea,
                 # if you don't know the length of lines
                 # that process ouput, this is a problem.
                 # I put 7 since I know the word "Testing" have
                 # 7 characters.
                 print os.read(fd, 7)

Maybe this is not exactly what you're looking for, but I think it give you a pretty good idea of what to do to solve your problem.

EDIT : I think I've just found what you need Streaming stdout from a Python subprocess in Python .