How to run multiple process simultaneously?

Question

I have a loop with highly time-consuming process and instead of waiting for each process to complete to move to next iteration, is it possible to run the process and just move to next iteration without waiting for it to complete?

Example : Given a text, the script should try to find the matching links from the Web and files from the local disk. Both return simply a list of links or paths.

for proc in (web_search, file_search):
   results = proc(text)
   yield from results

What I have as a solution is, using a timer while doing the job. And if the time exceeds the waiting time, the process should be moved to a tray and asked to work from there. Now I will go to next iteration and repeat the same. After my loop is over, I will collect the results from the process moved to the tray.

Answer 1

For simple cases, where the objective is to let each process run simultaneously, we can use Thread of threading module. So we can tackle the issue like this, we make each process as a Thread and ask it to put its results in a list or some other collection. The code is given below:

from threading import Thread

results = []

def add_to_collection(proc, args, collection):
    '''proc is the function, args are the arguments to pass to it.
       collection is our container (here it is the list results) for
       collecting results.'''
    result = proc(*args)
    collection.append(result)
    print("Completed":, proc)

# Now we do our time consuming tasks

for proc in (web_search, file_search):
    t = Thread(target=add_to_collection, args=(proc, ()))
    # We assume proc takes no arguments
    t.start()

For complex tasks, as mentioned in comments, its better to go with multiprocessing.pool.Pool .