Python 3 - Multiprocessing same function with different args

Question

In python 3, I am trying to run the same function with multiple arguments at the same time. I am using multiprocessing in Python 3.5.2, Anaconda 4.1.1 (64-bit), in Windows 7. I am getting the following error regarding spawn.py:

An attempt has been made to start a new process before the current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your child processes and you have forgotten to use the proper idiom in the main module:

if __name__ == '__main__':
    freeze_support()
    ...

The "freeze_support()" line can be omitted if the program is not going to be frozen to produce an executable.

My code is:

from multiprocessing import Process

#Function to be run
def savedatetime(password,rangestart,rangeend):
    print((password,rangestart,rangeend))

# Inputs to the function
passwords=['p0','p1','p2']
ranges=[(1,10),(10,20),(20,30)]

# Creating the list of processes to append
fileind=0
proc=[]
for rangeind in range(0,len(passwords)):    
    password=passwords[fileind]
    rangestart=ranges[fileind][0]
    rangeend=ranges[fileind][1]
    p = Process(target=savedatetime,args=(password,rangestart,rangeend))            
    proc.append(p)
    fileind=fileind+1
    # running sequentially to check that the function works
    savedatetime(password,rangestart,rangeend)

print(proc)

# Attempting to run simultaneously. This is where the error occurs:
for p in proc:
    p.start()
    p.join()

Could you please help me to fix my code so the multiple instances of the same function run simultaneously? Thanks!

Answer 1

You need a gatekeeper test to prevent your code from executing again when Windows simulates forking by reimporting your main module. It's why all main modules should control their "main-like" behavior with the if __name__ == '__main__': gatekeeper check.

The simplest solution is to take all the "main" code, indent it one level, and define a function named main that contains it.

The main function would be:

def main():

    # Inputs to the function
    passwords=['p0','p1','p2']
    ranges=[(1,10),(10,20),(20,30)]

    # Creating the list of processes to append
    ... omitted for brevity ...

    # Attempting to run simultaneously. This is where the error occurs:
    for p in proc:
        p.start()

    # Join after all processes started, so you actually get parallelism
    for p in proc:
        p.join()

Then just add the following to the end of your file:

if __name__ == '__main__':
    main()

The __name__ check prevents your main function from being re-executed when you spawn the worker processes.