How to properly encapsulate an asyncio Process

Question

I have a program executed in a subprocess. This program runs forever, reads a line from its stdin , processes it, and outputs a result on stdout . I have encapsulated it as follows:

class BrainProcess:
    def __init__(self, filepath):
        # starting the program in a subprocess
        self._process = asyncio.run(self.create_process(filepath))

        # check if the program could not be executed
        if self._process.returncode is not None:
            raise BrainException(f"Could not start process {filepath}")

    @staticmethod
    async def create_process(filepath):
        process = await sp.create_subprocess_exec(
            filepath, stdin=sp.PIPE, stdout=sp.PIPE, stderr=sp.PIPE)

        return process

    # destructor function
    def __del__(self):
        self._process.kill() # kill the program, since it never stops
        # waiting for the program to terminate
        # self._process.wait() is asynchronous so I use async.run() to execute it
        asyncio.run(self._process.wait())

    async def _send(self, msg):
        b = bytes(msg + '\n', "utf-8")

        self._process.stdin.write(b)
        await self._process.stdin.drain()

    async def _readline(self):
        return await self._process.stdout.readline()

    def send_start_cmd(self, size):
        asyncio.run(self._send(f"START {size}"))

        line = asyncio.run(self._readline())
        print(line)
        return line

From my understandingasyncio.run() is used to run asynchronous code in a synchronous context. That is why I use it at the following lines:

# in __init__
self._process = asyncio.run(self.create_process(filepath))

# in send_start_cmd
asyncio.run(self._send(f"START {size}"))
# ...
line = asyncio.run(self._readline())

# in __del__
asyncio.run(self._process.wait())

The first line seems to work properly (the process is created correctly), but the other throw exceptions that look like got Future <Future pending> attached to a different loop .

Code:

brain = BrainProcess("./test")
res = brain.send_start_cmd(20)
print(res)

So my questions are:

• What do these errors mean?
• How do I fix them?
• Did I use asyncio.run() correctly?
• Is there a better way to encapsulate the process to send and retrieve data to/from it without making my whole application use async / await ?

Answer 1

asyncio.run is meant to be used for running a body of async code, and producing a well-defined result. The most typical example is running the whole program:

async def main():
    # your application here

if __name__ == '__main__':
    asyncio.run(main())

Of couurse, asyncio.run is not limited to that usage, it is perfectly possible to call it multiple times - but it will create a fresh event loop each time. This means you won't be able to share async-specific objects (such as futures or objects that refer to them) between invocations - which is precisely what you tried to do. If you want to completely hide the fact that you're using async, why use asyncio.subprocess in the first place, wouldn't the regular subprocess do just as well?

The simplest fix is to avoid asyncio.run and just stick to the same event loop. For example:

_loop = asyncio.get_event_loop()

class BrainProcess:
    def __init__(self, filepath):
        # starting the program in a subprocess
        self._process = _loop.run_until_complete(self.create_process(filepath))
        ...
...

Is there a better way to encapsulate the process to send and retrieve data to/from it without making my whole application use async / await?

The idea is precisely for the whole application to use async/await, otherwise you won't be able to take advantage of asyncio - eg you won't be able to parallelize your async code.