Python read stdin from piped process and from Popen subprocess simultaneously

Question

I have a very specific problem where I am trying to read two streams simultaneously, one from a piped process to stdin, and one from a subprocess started with Popen. Call the program 'stream_compare.py'

Two threads are started: Thread 1 which goes and starts reading from stdin as such:

while True:
      line = sys.stdin.readline().rstrip('\n')
      # do something

And Thread 2 reads the following way:

cmd = ["./run.sh", "test.txt"]
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
while proc.poll() is None:
      for line in proc.stdout.readlines():
               # do something

Unfortunately, when starting the program, lines from the piped process that feed to stdin in Thread 1 seem to block subprocess.Popen from reading anything. Individually, I can get Thread 2 to run correctly when I don't pipe anything to 'stream_compare', otherwise it doesn't do anything.

Is there any way to read two streams like this simultaneously within the same program? Assuming one stream has to come from a piped process that's started remotely (from a different server), and one stream has to be started as a subprocess via Popen in the same directory as 'stream_compare'. Everything is being run on a linux machine.

Answer 1

this code:

for line in proc.stdout.readlines():

is waiting for stdout to close/process to be finished before starting to iterate on the lines because readlines() creates a list of lines. You have to loop on the line generator like this instead:

for line in proc.stdout:

(and in general, it's very rare to have a good reason to use readlines() , even with a "finite" file, when processing line by line, because it uses more memory for nothing)