Execute two functions concurrently using asyncio

Question

I currently have a setup where I open a subprocess and I have to read both stdout and stderr at the same time , so after invoking the subprocess I spawn a new thread for stdout and just handle stderr in the main thread.

# imports
from subprocess import Popen, PIPE
from threading import Thread


def handle_stdout(stdout):
    # ... do something with stdout,
    # not relevant to the question
    pass


def my_fn():
    proc = Popen([...], stdout=PIPE, stderr=PIPE)
    Thread(target=lambda: handle_stdout(proc.stdout)).start()
    # ... handle stderr
    print(proc.stderr.read())
    proc.wait()
    proc.kill()

my_fn()

Is there a way I can achieve the same thing using asyncio?

Answer 1

A thread-free asyncio version of your code might look something like this:

import asyncio
import asyncio.subprocess

async def handle_stdout(stdout):
    while True:
        line = await stdout.readline()  # Possibly adding .decode() to get str
        if not line:
            break
    # In 3.8 four lines above can be replaced with just:
    # while line := await stdout.readline():  # Yay walrus operator!
        # ... do stuff with line ...

async def my_fn():
    # Note: No list wrapping on command line arguments; all positional arguments are part of the command
    proc = await asyncio.create_subprocess_exec(..., stdout=asyncio.subprocess.PIPE, stderr=asyncio.subprocess.PIPE)
    stdout_task = asyncio.create_task(handle_stdout(proc.stdout))
    # ... handle stderr
    print(await proc.stderr.read())
    await stdout_task
    await proc.wait()

if  __name__ == '__main__':
    asyncio.run(my_fn())

The APIs are a little different, async functions are actually called when you make tasks from them (where threads must take uncalled functions), and you need to be careful to await all async actions taken, but it's not that different. The main issue is async 's viral nature; since you can only await in an async function, it's hard to call async code from non-async code (the other way around works fine, as long as the non-async code doesn't block for any reason). It makes async code bases largely incompatible with non- async stuff, and makes converting piecemeal nearly impossible, but for brand new code, it works fine.