如何并行启动功能，检查是否完成，并在python中启动一个新的function？

Question

I want to write a python code that does the following:

• At first, it starts, say, 3 processes (or threads, or whatever) in parallel.
• Then in a loop, python waits until any of the processes have finished (and returned some value)
• Then, the python code starts a new function

In the end, I want 3 processes always running in parallel, until all functions I need to run are run. Here is some pseudocode:

import time
import random
from multiprocessing import Process

# some random function which can have different execution time
def foo():
    time.sleep(random.randint(10) + 2)
    return 42

# Start 3 functions
p = []
p.append(Process(target=foo))
p.append(Process(target=foo))
p.append(Process(target=foo))

while(True):
    
    # wait until one of the processes has finished
    ???

    # then add a new process so that always 3 are running in parallel
    p.append(Process(target=foo))

I am pretty sure it is not clear what I want. Please ask.

Answer 1

What you really want is to start three processes and feed a queue with jobs that you want executed. Then there will only ever be three processes and when one is finished, it reads the next item from the queue and executes that:

import time
import random
from multiprocessing import Process, Queue

# some random function which can have different execution time
def foo(a):
    print('foo', a)
    time.sleep(random.randint(1, 10) + 2)
    print(a)
    return 42

def readQueue(q):
    while True:
        item = q.get()
        if item:
            f,*args = item
            f(*args)
        else:
            return
    
if __name__ == '__main__':
    q = Queue()
    for a in range(4):  # create 4 jobs
        q.put((foo, a))
    for _ in range(3):  # sentinel for 3 processes
        q.put(None)

    # Start 3 processes
    p = []
    p.append(Process(target=readQueue, args=(q,)))
    p.append(Process(target=readQueue, args=(q,)))
    p.append(Process(target=readQueue, args=(q,)))

    for j in p:
        j.start()
    #time.sleep(10)
    for j in p:
        j.join()

Answer 2

You can use the Pool of the multiprocessing module.

my_foos = [foo, foo, foo, foo]

def do_something(method):
     method()


from multiprocessing import Pool
with Pool(3) as p:
    p.map(do_something, my_foos)

The number 3 states the number of parallel jobs. map takes the inputs as arguments to the function do_something In your case do_something can be a function which calls the functions you want to be processed, which are passed as a list to inputs.