使用 python 并行执行 shell 脚本

Question

I have a task to optimize a process and for that, I need to orchestrate multiple shell scripts using python.

Let's assume I have 13 ksh scripts. I want to build the python script such that

1. some scripts run in parallel
2. some scripts are triggered after the completion of their predecessor scripts
3. some scripts are dependent on the output of their predecessor scripts and run according to a decision box.

I have summarized the above points in a flow diagram.

There is also a special requirement for failure as follows:

1. After completion of each shell it should create a particular flag
2. If a ksh (assume shell_script_3) fails then the program should stop and not create a flag
3. Once the job is fixed, when the program is re-run it should start from the ksh which is succeeding the failed ksh (shell_script_5)

I need help in designing a python script for the above requirement.

Answer 1

this can quickly become a complicated problem. but at the core, you have

1. a job definition that defines a jobs start condition (for example, time of day and/or completion of other jobs) execution
2. an execution engine that handles the running of the jobs, in parallel as needed
3. an event processor that tracks which jobs are awaiting time of or other conditions to start, and processes the status of execution engines. it would also be responsible for dispatching to the execution engine any new jobs, including providing any signalling information used by the execution plan for the decision tree.
4. lots of extra things like log access, controls around updating job definitions, cancelling, alerts, run-too-long events, restarting/cancelling jobs.

There are a lot of pitfalls in designing this, for example how do you detect and prevent loops in the job dependencies. there are many open source system out there for batch jobs, you should take a look.