python 3.4 sub process

Question

I am trying to control subprocess and control the stdout and stdin of this process. It worked great with python 2.7 but when using python 3 it fails to run.

I wrote the following code in python 2.7:

print("python 2.7:\n")
proc = subprocess.Popen(['cat -'],
                        shell = True,
                        stdin = subprocess.PIPE, 
                        stdout = subprocess.PIPE,
                        #stderr = subprocess.PIPE,
                        )
for i in range(10):
    proc.stdin.write('%d\n' % i)
    output = proc.stdout.readline()
    print (output.rstrip())

remainder = proc.communicate()[0]
remainder = remainder.replace("(","").replace(")","").replace("'","").split(",")[0]
print("-----------------")
print(remainder)

as I said, this code works fine, but when trying to write it in python 3.4 I fail to use the stdin.write properly.

print("python 3.4:\n")
proc = subprocess.Popen(['cat -'],
                        shell = True,
                        stdin = subprocess.PIPE, 
                        stdout = subprocess.PIPE,
                        stderr = subprocess.PIPE,
                        universal_newlines = True

                        )

for i in range(10):
    inp = str(i)
    proc.stdin.flush()
    proc.stdout.flush()
    proc.stdin.write(inp)

    output = proc.stdout.read()
    print (output.rstrip())

remainder = proc.communicate()[0]
remainder = remainder.replace("(","").replace(")","").replace("'","").split(",")[0]
print("-----------------")
print(remainder)

Answer 1

You almost there. It is a buffering problem:

• bufsize=0 (unbuffered) on Python 2.7 by default
• bufsize=-1 (fully buffered) on Python 3.4 (the default has been changed)

You should put proc.stdin.flush() after proc.stdin.write() (before the .write() call, the buffer is empty -- there is nothing to flush):

#!/usr/bin/env python3
from subprocess import Popen, PIPE

with Popen(['cat'], stdin=PIPE, stdout=PIPE, bufsize=1,
           universal_newlines=True) as process:
    for i in range(10):
        print(i, file=process.stdin, flush=True)
        output = process.stdout.readline()
        print(output, end='')

It is a complete code example.

Note: in general, unless the you know exactly how much you need to read (like in the example: one line input -> one line output exactly); the input/output might become desynchronized and the deadlock might happen eg, if the child produces less output than you expect then your process will block on .readline() while the child at the same time hangs awaiting for input from you. A more robust solution would work with child process' streams asynchronously (asyncio, threading, select).