Unable to get binary stdin/stdout in Subprocesses

Question

I've been trying to implement a subprocessing application using the Parallel Python ( pp ) module, which is using subprocess with subprocess.PIPE to pass serialized instructions and responses between master and worker processes. I've had intermittent issues with the workers' stdin.read() commands returning empty strings rather than the normal behavior of blocking until a command is received.

After a bit of research, I believe the cause is the worker processes' io streams are in text mode and attempting to pass pickled objects. They eventually get something that looks like an EOF and then automatically return empty. Looking in the pp source code, it even specifies the -u flag in its calling sequence, but the resulting workers' streams don't seem to be in binary mode, even though the calling Python interpreter is. The fix suggested here and elsewhere is to use the msvcrt module to change the io format to binary, but for some reason it doesn't seem to have any effect.

I made the following demo script. This is Python 2.7.12, 32-bit, in Windows 10:

#master.py
import subprocess

if __name__ == '__main__':

    foo = subprocess.Popen(
            args = ['pythonw.exe','-u','-m','worker'],
            stdin = subprocess.PIPE,
            stdout = subprocess.PIPE,
            stderr = subprocess.PIPE,
        )

    open('master_log.txt','a').write(str(foo.stdin) + '\n')
    open('master_log.txt','a').write(str(foo.stdout) + '\n')

...and in the same folder...

#worker.py
import sys, os, msvcrt

if __name__ == '__main__':
    open('worker_log.txt','a').write('Initial Properties\n')
    open('worker_log.txt','a').write(str(sys.argv[0]) + '\n')
    open('worker_log.txt','a').write(str(sys.stdin) + '\n')
    open('worker_log.txt','a').write(str(sys.stdout) + '\n')
    open('worker_log.txt','a').write('Applying msvcrt.setmode()\n')
    msvcrt.setmode(sys.stdin.fileno(),os.O_BINARY)
    msvcrt.setmode(sys.stdout.fileno(),os.O_BINARY)
    open('worker_log.txt','a').write(str(sys.stdin) + '\n')
    open('worker_log.txt','a').write(str(sys.stdout) + '\n')

From a windows command prompt:

python -u -m master.py

yields:

#master_log.txt
<open file '<fdopen>', mode 'wb' at 0x02FA06A8>
<open file '<fdopen>', mode 'rb' at 0x02FA0D30>

#worker_log.txt
Initial Properties
C:\Users\204040537\Documents\Python\pygtp_addin\worker.py
<open file '<stdin>', mode 'r' at 0x031BD020>
<open file '<stdout>', mode 'w' at 0x031BD078>
Applying msvcrt.setmode()
<open file '<stdin>', mode 'r' at 0x031BD020>
<open file '<stdout>', mode 'w' at 0x031BD078>

Suggestions? I'm at a loss for how else to coerce the subprocess IO streams to read binary.

Answer 1

Whelp, the resolution to this problem (appears) to be that a something related to an internal DLL my workers are using is causing my python interpreter to behave erratically. Interactive Python sessions exit after the line containing the offending command is complete, even if there are multiple statements on the line, eg

result = offending_command(some_data); print result

will run, and print the correct result, but the interpreter will immediately exit. Using the offending command in a python script and calling via execfile() will run normally, but the python interpreter immediately exits upon completion. Variations on some_data can repeatably cause or prevent this behavior.

My workers continue to run, as evidenced by ongoing error log entries, but their stdin pipe seems broken? Log messages indicate the stdin pipe is still open. In any case, this does not appear to be an issue with subprocess.