Running bash as a subprocess to Python using asyncio, but bash prompts are delayed

Question

I wish to control a long-running interactive Bash subprocess from Python's asyncio, send it commands one at a time, and receive results back from it.

The code fragment below works perfectly well in Python 3.7.0, Darwin Kernel Version 16.7.0, except that Bash prompts do not appear immediately on stderr , but appear to "queue up" until something else writes to stderr .

This is a problem because the original program needs to receive the Bash prompt to know that the previous command has finished.

from asyncio.subprocess import PIPE
import asyncio


async def run():
    proc = await asyncio.create_subprocess_exec(
        '/bin/bash', '-i', stdin=PIPE, stdout=PIPE, stderr=PIPE
    )

    async def read(stream):
        message = 'E' if stream is proc.stderr else 'O'
        while True:
            line = await stream.readline()
            if line:
                print(message, line)
            else:
                break

    async def write():
        for command in (b'echo PS1=$PS1', b'ls sub.py', b'ls DOESNT-EXIST'):
            proc.stdin.write(command + b'\n')
            await proc.stdin.drain()
            await asyncio.sleep(0.01)  # TODO: need instead to wait for prompt

    await asyncio.gather(
        read(proc.stderr),
        read(proc.stdout),
        write(),
    )


asyncio.run(run())

Results:

E b'bash: no job control in this shell\n'
O b'PS1=\\u@\\h:\\w$\n'
O b'sub.py\n'
E b'tom@bantam:/code/test/python$ tom@bantam:/code/test/python$ tom@bantam:/code/test/python$ ls: DOESNT-EXIST: No such file or directory\n'

Note that the three prompts all come out together at the end, and only once an error was deliberately caused. The desired behavior would of course be for the prompts to appear immediately as they occurred.

Using proc.stderr.read() instead of proc.stderr.read() results in more code but just the same results.

I'm a little surprised to see that bash: no job control in this shell message appear in stderr , because I am running bash -i and because $PS1 is set, and I wonder if that has something to do with the issue but haven't been able to take that further.

Answer 1

This held me up for half a day, but once I finished writing the question up, it took me ten minutes to come up with a workaround.

If I modify the prompt so it ends with a \n , then proc.stderr is in fact flushed, and everything works absolutely perfectly.