如何将管道中的消息从一个进程流传输到另一个进程？

Question

I have 2 python (2.7) processes.
The parent process needs to send rows of text to a child process, and the child process should process them as they come in (not wait for the parent process to finish).

I have this code which doesn't work:

# Sender
import subprocess

process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)

try:
    while True:
        process.stdin.write(msg + '\n')  # 'msg' is a changing string
        # process.stdin.flush() <-- commented out since it doesn't help
except KeyboardInterrupt:
    process.stdin.close()
    process.wait()

And the child process:

# Receiver
import sys

for line in sys.stdin:
    print line.strip()

The problem is that the child process waits until the parent process exits before it prints out the messages.

What I'm trying to achieve is a child process that processes the messages as soon as they are written to the pipe.

Answer 1

Try adding a process.stdin.flush() after your process.stdin.write() . That way you actually send the string to the other process. What you're suffering from here is your kernel caching everything you write. It does this to be more efficient when actually sending the data to the other process. flush force the kernel to send your data regardless of how full the kernel's buffer is.

I tried your code as such:

# Sender
import subprocess                                                                                                                                                                                           

process = subprocess.Popen(['python', 'child.py'], bufsize=1, stdin=subprocess.PIPE)
msg = "This is my message"

try:
    while True:
        process.stdin.write(msg + '\n')  # 'msg' is a changing string
        process.stdin.flush() # This code works well for me regardless of the presence of this line
except KeyboardInterrupt:
    process.stdin.close()
    process.wait()

# Receiver
import sys

for line in sys.stdin:
    print line.strip()

With "works well" here i mean that i get "This is my message" printed as fast as the computer can perform. I'm trying this in Python 2.7.12 for the record.

Answer 2

The story of how buffering works for sys.stdin and sys.stdout has made me cry more than once. A similar problem is discussed in Setting smaller buffer size for sys.stdin? .

As to your specific problem, I suggest you change your child to use sys.stdin.readline() instead of iterating over sys.stdin . The former somewhat "buffers less" :)

while True:
    line = sys.stdin.readline()
    if not line: break
    print (line.strip())

In the parent, you'll likely either need to set bufsize=0 in your call to Popen (making your pipe completely unbuffered), or you'll need the process.stdin.flush() line, as Patrik suggests. I'd opt for the latter.

Tested on Python 2.7.14 on Windows 10 64bit.