python：nonblocking subprocess，检查stdout

Question

Ok so the problem I'm trying to solve is this:

I need to run a program with some flags set, check on its progress and report back to a server. So I need my script to avoid blocking while the program executes, but I also need to be able to read the output. Unfortunately, I don't think any of the methods available from Popen will read the output without blocking. I tried the following, which is a bit hack-y (are we allowed to read and write to the same file from two different objects?)

import time
import subprocess
from subprocess import *
with open("stdout.txt", "wb") as outf:
    with open("stderr.txt", "wb") as errf:
        command = ['Path\\To\\Program.exe', 'para', 'met', 'ers']
        p = subprocess.Popen(command, stdout=outf, stderr=errf)
        isdone = False
        while not isdone :
            with open("stdout.txt", "rb") as readoutf: #this feels wrong
                for line in readoutf:
                    print(line)
            print("waiting...\\r\\n")
            if(p.poll() != None) :
                done = True
            time.sleep(1)
        output = p.communicate()[0]    
        print(output)

Unfortunately, Popen doesn't seem to write to my file until after the command terminates.

Does anyone know of a way to do this? I'm not dedicated to using python, but I do need to send POST requests to a server in the same script, so python seemed like an easier choice than, say, shell scripting.

Thanks! Will

Answer 1

Basically you have 3 options:

1. Use threading to read in another thread without blocking the main thread.
2. select on stdout, stderr instead of communicate . This way you can read just when data is available and avoid blocking.
3. Let a library solve this, twisted is a obvious choice .

Answer 2

You can use twisted library for this use case. I think it will be great for that

http://www.cs.lth.se/EDA046/assignments/assignment4/twisted/listings/process/quotes.py

documentation : http://www.cs.lth.se/EDA046/assignments/assignment4/twisted/process.html