Execute and monitor multiple instances of external program in Python

Question

Main program is like this:

PREPARE PARAMETERS FOR CHILD PROCESSES
subprocess.Popen('python child.py param=example1'.split(' '))
subprocess.Popen('python child.py param=example2'.split(' '))
...

How to make main program to monitor each instances of child process it launched and restart it with its corresponding parameters if it's not running.

The purpose for keep multiple instances of child process running instead of implementing a multi-thread architect within main process is to utilize as much CPU and database throughputs as possible.

Answer 1

Keep a dict with the .pid s of the child processes as keys, and the commandlines to restart them as corresponding values. ie:

childid = []
for cmdline in cmdlines:
  p = subprocess.Popen(cmdline.split())
  childid[p.pid] = cmdline

os.wait will return whenever any child process terminates: it gives you (pid, exitstatus) of the child. So just restart appropriately and maintain childid . ie:

while mustcontinue:
  pid, exitstat = os.wait()
  cmdline = childid.pop(pid)
  p = subprocess.Popen(cmdline.split())
  childid[p.pid] = cmdline

Presumably you have some criteria for when this infinite loop ends, I just used mustcontinue as the name for those criteria here;-).