如何在 python 中广泛和/或交替地从 stdout 和/或 stderr 读取并写入进程的 stdin？

Question

I searched here a lot of different questions and answers, but I did not found a general approach for:

• reading from stdout and stderr, what is available - until the last byte (currently!) available (even if it is not a \n)
• depending upon the read information write something to stdin
• the command line tool will react to this stdin input and write (much later) something/nothing
• start from beginning - or leave loop if process has finished, capturing its return code

Most examples here findable write only ONCE to stdin and read only ONCE (before/afterwards) from stdout and/or stderr. My intention is to "weave" reading from stdout and/or stderr and writing to stdin! Here an example:

• starting a command line tool (finally with parameters) - eg python3.exe
• reading always from the stdout and stderr
• eg read everything and after reading >>> from stdout
• write print('Hello World.')\n
• eg read everything ( Hello World.\n )and after reading >>> from stdout
• write x = [6, 0]\n
• eg read everything and after reading >>> from stdout
• write y = x[0] / x[1]\n
• eg read everything (... ZeroDivisionError: division by zero on stdout/stderr)
• ...

I tried to solve it with this found internet example (after other failed attempts):

# Example #27
# of https://www.programcreek.com/python/example/85342/asyncio.create_subprocess_shell
# Source Project: Python-Journey-from-Novice-to-Expert   Author: PacktPublishing   File: 07_processes.py    License: MIT License    5 votes vote downvote up
import asyncio
import sys

async def read_from_pipe(pipe, buf, timeout_sec):
    while True:
        try:
            pipe_byte = await asyncio.wait_for(pipe.read(1), timeout_sec)
        except asyncio.TimeoutError:
            break
        else:
            if len(pipe_byte) == 1:
                buf.append(pipe_byte[0])
            else:
                pipe_byte == b'\n' # in case of end of file: fake end of line 
            if pipe_byte == b'\n':
                return len(buf)

async def run_script(version):
    process = await asyncio.create_subprocess_shell(
        r'C:\Programs\Python\Python38-32\python.exe',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )

    if version == 0:
        # Write a simple Python script to the interpreter
        process.stdin.write(b'\n'.join((
            b'import math',
            b'x = 2 ** 8',
            b'y = math.sqrt(x)',
            b'z = math.sqrt(y)',
            b'print("x: %d" % x)',
            b'print("y: %d" % y)',
            b'print("z: %d" % z)',
            b'for i in range(int(z)):',
            b'    print("i: %d" % i)',
        )))
        # Make sure the stdin is flushed asynchronously
        await process.stdin.drain()
        # And send the end of file so the Python interpreter will
        # start processing the input. Without this the process will
        # stall forever.
        process.stdin.write_eof()

        # Fetch the lines from the stdout asynchronously
        async for out in process.stdout:
            # Decode the output from bytes and strip the whitespace
            # (newline) at the right
            print(out.decode('utf-8').rstrip())

        # Wait for the process to exit
        await process.wait() 
    elif version == 1:
        cmds = [b'import math',
          b'x = 2 ** 8',
          b'y = math.sqrt(x)',
          b'z = math.sqrt(y)',
          # b'q = z / 0',
          b'print("x: %d" % x)',
          b'print("y: %d" % y)',
          b'print("z: %d" % z)',
          b'for i in range(int(z)):',
          b'    print("i: %d" % i)',
          b'exit(0)',
        ]
        idx = 0
        while True:
            stdout_buf = bytearray(b'')           
            out_read = await read_from_pipe(process.stdout, stdout_buf, 0.5)
            print(f'stdout[{out_read}]: {stdout_buf.decode("ascii")}\n') if out_read else None
            stderr_buf = bytearray(b'')             
            err_read = await read_from_pipe(process.stderr, stderr_buf, 0.5)
            print(f'stderr[{err_read}]: {stdout_buf.decode("ascii")}\n') if err_read else None
            if idx < len(cmds):
                current_cmd = cmds[idx].decode('ascii')
                print(f'writing command at index {idx}: "{current_cmd}"')
                process.stdin.write(cmds[idx])
                process.stdin.write(b'\n')
                await process.stdin.drain()
                process.stdin.write_eof() # tried with/without this line, afterwards program hangs
                idx += 1
            else:
                break
        await process.wait()             

if sys.platform == "win32":
    codepage = 'cp437'
    loop = asyncio.ProactorEventLoop() # For subprocess' pipes on Windows
    asyncio.set_event_loop(loop)
else:
    codepage = 'utf-8'
    loop = asyncio.get_event_loop()

version = 1 # version = 0 runs but is not alternatingly reading stdout/stderr and writing to stdin!
returncode = loop.run_until_complete(run_script(1))
print(f'done with return code = {returncode}.')

Currently it doesn't read anything from the stdout and stderr . And after the entries in cmds are written, program hangs too. Finally it should run under linux.

How do I write the program correctly?

Is python3.exe a "too special" command line tool and is the root cause of these problems?

Hint: This example and the solution do not have to be performant at all. The intended command line tool to control is quite slow (overall execution 20 s to 20 min). Multithreading and multiprocessing is not really required, if not needed for a (simplified) working solution.

Answer 1

I found out that python3.exe is a bit too special to control. I better use eg cmd /S on windows (I read /bin/bash for Linux) - this works now:

# Example #27
# of https://www.programcreek.com/python/example/85342/asyncio.create_subprocess_shell
# Source Project: Python-Journey-from-Novice-to-Expert   Author: PacktPublishing   File: 07_processes.py    License: MIT License    5 votes vote downvote up
import asyncio
import sys

async def read_from_pipe(pipe, buf, timeout_sec):
    while True:
        try:
            pipe_byte = await asyncio.wait_for(pipe.read(1), timeout_sec)
        except asyncio.TimeoutError:
            return len(buf) # no more bytes available currently on that pipe
        else:
            if len(pipe_byte) == 1:
                buf.append(pipe_byte[0])
            else:
                return len(buf) # end of pipe reached

async def run_script():
    process = await asyncio.create_subprocess_shell(
        'cmd /S',
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
        stdin=asyncio.subprocess.PIPE,
    )

    cmds = [b'dir P*C*S*.*',
    b'echo %temp%',
    b'exit']
    idx = 0
    while True:
        stdout_buf = bytearray(b'')           
        out_read = await read_from_pipe(process.stdout, stdout_buf, 0.5)
        print(f'stdout[{out_read}]: {stdout_buf.decode("ascii")}\n') if out_read else None
        stderr_buf = bytearray(b'')             
        err_read = await read_from_pipe(process.stderr, stderr_buf, 0.5)
        print(f'stderr[{err_read}]: {stdout_buf.decode("ascii")}\n') if err_read else None
        if idx < len(cmds):
            current_cmd = cmds[idx].decode('ascii')
            print(f'writing command at index {idx}: "{current_cmd}"')
            process.stdin.write(cmds[idx])
            process.stdin.write(b'\n')
            await process.stdin.drain()
            idx += 1
        else:
            pass
        if process.returncode is not None:
            print(f'return code = {process.returncode}')
            return process.returncode

if sys.platform == "win32":
    codepage = 'cp437'
    loop = asyncio.ProactorEventLoop() # For subprocess' pipes on Windows
    asyncio.set_event_loop(loop)
else:
    codepage = 'utf-8'
    loop = asyncio.get_event_loop()

returncode = loop.run_until_complete(run_script())
print(f'done with return code = {returncode}.')

The output is on my computer:

PS C:\Git\ownPythonRepository\Python\CliTap>  c:; cd 'c:\Git\ownPythonRepository\Python\CliTap'; & 'C:\Programs\Python\Python38-32\python.exe' 'c:\Users\BitLauncher\.vscode\extensions\ms-python.python-2022.14.0\pythonFiles\lib\python\debugpy\adapter/../..\debugpy\launcher' '63136' '--' 'c:\Git\ownPythonRepository\Python\CliTap\PythonConsoleSandbox.py' 
stdout[137]: Microsoft Windows [Version 10.0.11111.2222]
(c) Microsoft Corporation. All rights reserved.

C:\Git\ownPythonRepository\Python\CliTap>

 stdout[340]: dir P*C*S*.*
 Volume in drive C is What
 Volume Serial Number is 9999-9999

 Directory of C:\Git\ownPythonRepository\Python\CliTap

2022-09-26  23:52             2,365 PythonConsoleSandbox.py
               1 File(s)          2,365 bytes
               0 Dir(s)  99,999,999,999 bytes free

C:\Git\ownPythonRepository\Python\CliTap>
writing command at index 1: "echo %temp%"
stdout[93]: echo %temp%
C:\Users\BitLau~1\AppData\Local\Temp

C:\Git\ownPythonRepository\Python\CliTap>

writing command at index 2: "exit"
stdout[5]: exit


return code = 1
done with return code = 1.
PS C:\Git\ownPythonRepository\Python\CliTap> 

That's it - now I will be able to write depending upon stdout and/or stderr specific commands to stdin... great. Later I can improve it by multithreading:-) if needed.