python: multiprocessing.Pipe and redirecting stderr on Windows

Question

I've a main process where I open a multiprocessing.Pipe(False) and send the writing end to a worker Process. Then, in the worker process, I run a Java program using subprocces.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE) . I need to redirect the error of this subprocess to the writing end of multiprocessing.Pipe

For this I referred to this answer by Ilija as this is exactly what I want to achieve, but on my machine(Windows), it throws OSError: [Errno 9] Bad file descriptor

Machine details:

OS - Windows 10 (64bit)

Python version - 3.7.4

Code:

Method 1 ( Ilija's answer )

def worker(w_conn):
    os.dup2(w_conn.fileno(), 2)
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
    sp.wait()
    w_conn.close()


def main():
    r_conn, w_conn = multiprocessing.Pipe(False)
    process = multiprocessing.Process(target=worker, args=(w_conn,))
    process.start()
    
    while not r_conn.poll() and not w_conn.closed:
        # Do stuff
    else:
        # Read error from r_conn, and handle it
    
    r_conn.close()
    process.join()

if __name__=='__main__':
    main()

Error:

Process Process-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\Desktop\Workspace\Error.py", line 14, in worker
    os.dup2(w_conn.fileno(), 2)
OSError: [Errno 9] Bad file descriptor

Method 2: In worker function, sending w_conn as argument to Popen

def worker(w_conn):
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
    sp.wait()
    w_conn.close()

Error:

Process Process-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 297, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\lib\multiprocessing\process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Users\User\Desktop\Workspace\Error.py", line 13, in worker
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=w_conn)
  File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 728, in __init__
    errread, errwrite) = self._get_handles(stdin, stdout, stderr)
  File "C:\ProgramData\Anaconda3\lib\subprocess.py", line 1077, in _get_handles
    errwrite = msvcrt.get_osfhandle(stderr.fileno())
OSError: [Errno 9] Bad file descriptor

Is there any workaround/alternate method to achive this on Windows?

Answer 1

I still don't know why "Method 1" is not working. Any information regarding this will be appreciated.

"Method 2" is wrong altogether as we can't use Connection object (returned by multiprocessing.Pipe() ) as a file handle in subprocess.Popen .

What works is checking for data in stderr of subprocess sp and sending the data through w_conn to main process.

def worker(w_conn):
    sp = subprocess.Popen(['java', 'myprogram'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
    sp.wait()
    if sp.stderr.seek(0, io.SEEK_END)>0:
        w_conn.send(sp.stderr.read())
    w_conn.close()