subprocess Popen.process.stdout.readline hangs randomly

Question

I am trying to start a 3rd party program from python and send messages to it, reading the answer. I do not want to restart the 3rd party program after every message.

So what I have simplifies to this code:

process = subprocess.Popen([executable, "-b", paramValue], 
            stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True, bufsize=1)
# clear some startup message, which for some reason is written into stderr
process.stderr.readline()

while 100k+ loops:
    process.stdin.write(txtInputLine)
    resultLine = process.stdout.readline().strip()
    processData(txtInputLine, resultLine)

After some random amount of time, often around 30 minutes after the loop has started the program hangs and stops working. It appears readline() just hands and stops working entirely.

How can I prevent this? I would be fine with having to restart the process (maybe using some kind of timeout?) if it fails or something like that.

But I cannot even find where readline() is documented, it seems it is just mentioned in the subprocess docs, without a full documentation on the function and google has a hard time avoiding threads of people who have problems with missing newlines and other problems that prevent readline from working in a much more deterministic manner.

Answer 1

The problem was some outputs on stderr that were never read and ended up blocking everything. Setting stderr to write to devnull solved the issue.