如何在Python中运行一系列异步Shell脚本

Question

I have a nodejs script using parallel async calls with callbacks in order to execute a series of shell scripts in the order they need to be run. (Data analysis tasks which are dependent on the numbers being crunched by the previous steps)

pseudo-code nodejs example:

async.parallel([
  async.apply('/usr/local/bin/foo some/path/to/some/script.sh 1'),
  async.apply('/usr/local/bin/foo some/path/to/some/script.sh 2'),
], function () {
  async.parallel([async.apply('/usr/local/bin/foo some/path/to/some/script.sh 3')];
});

So it would run the first two at the same time and when they've both completed, run the next ones. In reality this is many levels deep. I'm looking to replace this with a Python version but am struggling to figure out which modules/packages to leverage to make it happen.

Searches have brought up numerous possible ways to do this in Python which is making it a bit tricky to suss out which one to go for. Asyncio, subprocess, etc.

Answer 1

You could use multiprocessing and subprocess modules

from multiprocessing import Pool
from subprocess import call

Pool(2).map(call, [
    ["/usr/local/bin/foo some/path/to/some/script.sh", "1"],
    ["/usr/local/bin/foo some/path/to/some/script.sh", "2"]
])

call(["/usr/local/bin/foo some/path/to/some/script.sh", "3"])

Pool.map will call a function (the first argument) with each element of the array (the second argument) asynchronously. Once all of the calls are finished it will return a list with results (not used in this case) and code will continue executing as usual. So the third script will run once the first two are completed