Run same function in parallel with different parameters and know which parallel run has ended in python

Question

I have an execution function which should be run on more than one device parallelly. And I need to be calling a different function on completing each parallel run. I can't wait for all the parallel call to complete as it takes very different time according to the parameter passed.

    def func1(device, arg1, arg2):
        # do something

    for device in devices:
       # Call func1 with different arguments in parallel
       # If one of the parallel is finished call func2(arg, arg1, arg2) with different arguments.

How can I do this in Python?

Answer 1

See this code sample which uses the future.ThreadPoolExecutor for parallel execution.

from concurrent.futures import ThreadPoolExecutor, Future, as_completed

devices = []


def func1(device, arg1, arg2):
    pass


def do_after_func1(func1_result: Future):
    identifier = func1_result.arg
    result = func1_result.result()
    # do what ever
    pass


device_executors = ThreadPoolExecutor(max_workers=20)
args = []  # can be anything
futures = []
for device in devices:
    tracker = device_executors.submit(func1, *args)
    tracker.arg = "some-identififcation-if-you-need"
    tracker.add_done_callback(do_after_func1)
    futures.append(tracker)

You can submit anything to be executed in parallel as a callable and the result can be funneled to a call back function do_after_func1 . You can decide what to be called afterwards here. All this happens in parallel.

You can also make use of the ProcessPoolExecutor if you think multiple processes is needed.

You can learn more from the official documentation.