Use subprocess.communicate() to pipe stdin without waiting for process

Question

I need to launch a process and pipe a string in to stdin, which I am currently doing like this:

proc = subprocess.Popen(["MyCommandHere"], stdin=subprocess.PIPE)
proc.communicate(input=bytes(my_str_input + "\n", "ascii"))

The problem is that when I use subprocess.communicate() it is a blocking call, waiting for the process to exit. I do not want to wait.

Is there some way to get communicate() to not block, or some other way to pipe my input? I am asking about non-blocking writes, not non-blocking reads.

Answer 1

Two obvious options:

1. Use a separate thread or process
2. Feed stdin from a temporary file

Option 1:

import threading

def send_data_to(proc, inp):
    proc.communicate(inp)

proc = subprocess.Popen(["MyCommandHere"], stdin=subprocess.PIPE)
threading.Thread(target=send_data_to, args=(proc, bytes(my_str_input + "\n", "ascii"))).start()

Option 2:

import tempfile

with tempfile.TemporaryFile() as tf:
    tf.write(bytes(my_str_input + "\n", "ascii"))
    tf.flush()
    tf.seek(0)  # Might not be needed
    proc = subprocess.Popen(["MyCommandHere"], stdin=tf)

The write to the temporary file can block, but usually temporary files are optimized by the OS to minimize writes to disk when possible; if the process might take some time to finish, you might block too long piping directly, but the small blocking for writing out the data won't matter. Even though Python closes the temporary file when the with block exits (which would normally cause it to be deleted), the process maintains a handle to it, preventing it from being cleaned up until the process completes.

Note: All of this assumes the process might not consume your input completely immediately on launch. If the process basically reads the input immediately, then does all its work, you can simplify to just:

proc.stdin.write(bytes(my_str_input + "\n", "ascii"))
proc.stdin.close()  # Ensures the process knows nothing else is coming

This just risks blocking if the process consumes the input a little at a time, and the input is larger than the pipe buffers (so you can't write it all at once).

Answer 2

Take a look at the docs on Popen.stdin . It's just a standard writable object (and, in most cases, a standard file handle anyway), so you can do:

proc.stdin.write(bytes(...))

To write data to stdin without needing to wait for the subprocess to complete.