How to run 10 python programs simultaneously?

Question

I have a_1.py ~ a_10.py

I want to run 10 python programs in parallel.

I tried:

from multiprocessing import Process
import os

def info(title):
    I want to execute python program

def f(name):
    for i in range(1, 11):
        subprocess.Popen(['python3', f'a_{i}.py'])


if __name__ == '__main__':
    info('main line')
    p = Process(target=f)
    p.start()
    p.join()

but it doesn't work

How do I solve this?

Answer 1

I would suggest using the subprocess module instead of multiprocessing :

import os
import subprocess
import sys


MAX_SUB_PROCESSES = 10

def info(title):
    print(title, flush=True)

if __name__ == '__main__':
    info('main line')

    # Create a list of subprocesses.
    processes = []
    for i in range(1, MAX_SUB_PROCESSES+1):
        pgm_path = f'a_{i}.py'  # Path to Python program.
        command = f'"{sys.executable}" "{pgm_path}" "{os.path.basename(pgm_path)}"'
        process = subprocess.Popen(command, bufsize=0)
        processes.append(process)

    # Wait for all of them to finish.
    for process in processes:
        process.wait()

    print('Done')

Answer 2

If you just need to call 10 external py scripts ( a_1.py ~ a_10.py ) as a separate processes - use subprocess.Popen class:

import subprocess, sys

for i in range(1, 11):
    subprocess.Popen(['python3', f'a_{i}.py'])

# sys.exit()   # optional

---

It's worth to look at a rich subprocess.Popen signature (you may find some useful params/options)

Answer 3

You can use a multiprocessing pool to run them concurrently.

import multiprocessing as mp

def worker(module_name):
    """ Executes a module externally with python """
    __import__(module_name)
    return

if __name__ == "__main__":
    max_processes = 5
    module_names = [f"a_{i}" for i in range(1, 11)]
    print(module_names)
    with mp.Pool(max_processes) as pool:
        pool.map(worker, module_names)

The max_processes variable is the maximum number of workers to have working at any given time. In other words, its the number of processes spawned by your program. The pool.map(worker, module_names) uses the available processes and calls worker on each item in your module_names list. We don't include the .py because we're running the module by importing it.

Note: This might not work if the code you want to run in your modules is contained inside if __name__ == "__main__" blocks. If that is the case, then my recommendation would be to move all the code in the if __name__ == "__main__" blocks of the a_{} modules into a main function. Additionally, you would have to change the worker to something like:

def worker(module_name):
    module = __import__(module_name) # Kind of like 'import module_name as module'
    module.main()
    return