Python multiprocessing: all processes finish in 5 seconds but 10 additional seconds needed for the program to return to reading the main script

Question

I recently started learning Python multiprocessing. Every time, after all processes have finished, python needs an additional second to close one process (so if I have 10 open it will take 10s) and return to the rest of the script. So the issue is that a simple.sleep() program takes more time using multiprocessing than it would running a normal, linear program. Code:

def do_stuff(seconds):
    print("Sleeping for {} seconds...".format(seconds))
    time.sleep(seconds)
    print("Done sleeping... yawn...")



if __name__ == "__main__":
    start = time.perf_counter()

    process_list = []
    for _ in range(10):
        p = multiprocessing.Process(target=do_stuff, args=[5])
        p.start()
        process_list.append(p)

    for process in process_list:
        process.join()


    end = time.perf_counter()
    print(end-start)

The output looks like this:

Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...

Up until this point the program took about 5 seconds to finish all 10 processes

15.2574748

But then it waits 10 more seconds to finish the code and return the time needed for the program to finish How do I get around this issue? Thanks.

Answer 1

Your code seems okey. However, if it is not working for your OS, you could try defining the maximum number of threads you would like to have running at the same time. In the following example it is set to 10.

import time
import multiprocessing

def do_stuff(seconds):
    print("Sleeping for {} seconds...".format(seconds))
    time.sleep(seconds)
    print("Done sleeping... yawn...")

def chunks(l, n):
    for i in range(0, len(l), n):
        yield l[i:i + n]    

numberOfThreads = 10

if __name__ == "__main__":
    start = time.perf_counter()

    process_list = []
    for _ in range(10):
        p = multiprocessing.Process(target=do_stuff, args=[5])
        process_list.append(p)

    for i in chunks(process_list,numberOfThreads):
        for j in i:
            j.start()
        for j in i:
            j.join()


    end = time.perf_counter()
    print(end-start)

result:

Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Sleeping for 5 seconds...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
Done sleeping... yawn...
5.6765913999999995