How to asynchronously call a shell script from Python?

Question

I have a shell script which does some processing over the string passed and then writes it to a file. However I don't want my function foo() to wait for it to complete the operation. How do I call process(msg) and then move on the with the execution of {code block 2} without waiting for process(msg) to complete execution?

def process(msg):
    subprocess.call(['sh', './process.sh', msg])

def foo():
    # {code block 1}
    process(msg)
    # {code block 2}

foo() will be called from another function, almost once or twice per second.

Answer 1

What you need is https://docs.python.org/3/library/os.html#os.fork ie os.fork() that way you can spawn a child which can outlive the parent process which can be later claimed by systemd on Linux. I have no clue about Windows.

Answer 2

Just for completeness: Python's asyncio offers a high level interface for doing just that: https://docs.python.org/3.9/library/asyncio-subprocess.html#subprocesses

Example from documentation:

import asyncio

async def run(cmd):
    proc = await asyncio.create_subprocess_shell(
        cmd,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE)

    stdout, stderr = await proc.communicate()

    print(f'[{cmd!r} exited with {proc.returncode}]')
    if stdout:
        print(f'[stdout]\n{stdout.decode()}')
    if stderr:
        print(f'[stderr]\n{stderr.decode()}')

asyncio.run(run('ls /zzz'))

Answer 3

subprocess.call() and subprocess.run() creates a process, waits for it to finish, and returns a CompletedProcess object.

subprocess.Popen() creates a process and returns it. It is used under the hood of the previous functions. You can then wait for the process to finish, send it messages, or whatever else you want to do with it. The arguments are mostly the same as to call or run .

https://docs.python.org/3/library/subprocess.html

As a bit of elaboration, Popen is the python implementation of using the os to start a new process. os.fork() is a lower level that doesn't actually do what we want here, that would spawn another instance of the python interpreter with the same memory state as the current one. If you wanted to use the lower level syscall, os.spawn is closer to subprocess.run than os.fork .

To verify that Popen is doing what you want, this test program will pring "returncode = None", then wait 5 seconds, and print "returncode = 0"

from subprocess import Popen

p = Popen(["sleep", "5"])
print("started the proc")  # this will print immediately
p.poll()  # this checks if the process is done but does not block
print(f"p returncode = {p.returncode}")
p.wait()  # this blocks until the process exits
print(f"p returncode = {p.returncode}")