Python3.9 multiprocessing results in scoping error on MacOS

Question

I've encountered an error running an original script on iMac (2011, macOS High Sierra, Intel Core i7 2600, Python3.9.2), so reproduced behavior with simple code below:

from multiprocessing import Pool
import time


def raise_to_power(number):
   number = number ** power
   return number


if __name__ == '__main__':
   start_time = time.time()
   threads = 8
   power = 10
   numbers = [x for x in range(1,1000000)]
   chunksize, rem = divmod(len(numbers), threads)
   if rem: chunksize += 1
   with Pool(threads) as p:
      Product = p.map(raise_to_power,numbers,chunksize)
   print(time.time() - start_time)

The Output on iMac gives me:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker
result = (True, func(*args, **kwds))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar
return list(map(*args))
File "/Users/yn/PycharmProjects/test-multiprocessing/test-multiprocessing_thinkpad2.py", line 6, in raise_to_power
number = number ** power
NameError: name 'power' is not defined
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/Users/yn/PycharmProjects/test-multiprocessing/test-multiprocessing_thinkpad2.py", line 18, in <module>
Product = p.map(raise_to_power,numbers,chunksize)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map
return self._map_async(func, iterable, mapstar, chunksize).get()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get
raise self._value
NameError: name 'power' is not defined

The same code runs correctly on Thinkpad t420 (Linux Mint 20, core i5 2540m, Python3.9.2) with an execution time of ~0.6-0.7 seconds.

I rewrote the code to overcome scoping error:

from multiprocessing import Pool
import time
from itertools import product


def raise_to_power(number, power):
   number = number ** power
   return number


if __name__ == '__main__':
   start_time = time.time()
   threads = 8
   power = 10
   numbers = [x for x in range(1,1000000)]
   chunksize, rem = divmod(len(numbers), threads)
   if rem: chunksize += 1
   with Pool(threads) as p:
      Product = p.starmap(raise_to_power,product(numbers,[power]),chunksize)
   print(time.time() - start_time)

Now it runs correctly on both systems (and iMac is only 0.15 seconds faster on average), but I haven't found any clues in multiprocessing documentation about the described behavior. Could you point it out, please?

Answer 1

Let's look at your original approach where power was undefined. First, a couple of comments. You are calling worker function multiply_by , but I do not see where that is defined either. I do see, however, function raise_to_power . The error you should have gotten is that multiply_by is not defined. So this is a bit puzzling. Second, I see that you are computing a chunksize rather than using the default function for doing this which would compute a value roughly 1/4 the size you compute. Larger chunksizes mean fewer memory transfers (good) but could result in processors ending up being idle if they do not process their tasks at the same rate (bad), which granted is not that likely in your case. I can see having your own function for calculating the chunksize since the function used by the map method must convert its iterable argument to a list if necessary in order to get its length and if the iterable is very large, this could be very memory-inefficient. But you have already converted a range to a list so you haven't taken advantage of the opportunity to save memory by doing your own calculation.

As Mark Satchell indicated, the simple solution for your case would be just to make power a global. But let's consider the general case. If your platform is Windows or any platform that uses spawn to create new processes (I am guessing this well might be the case based on your use of if __name__ == '__main__': governing the code that creates new processes), then any code that is at the global scope will be executed for every new process created. This is not an issue for a statement like power = 10 . But if power required far more complicated code to initialize its value, it would be inefficient to re-execute this code over and over again for each process in the pool of processes. Or consider the case where power was a very large array. It would be perhaps too costly to create instances of this array in each sub-process's memory space. Then what is required is a single instance of the array in shared memory.

There is a mechanism for initializing global variables for each sub-process in a pool by using the initializer and initargs arguments when creating the Pool instance. I have also made an additional change to save memory by taking advantage of the fact that you are using your own chunksize calculation:

from multiprocessing import Pool
import time


def init_pool(the_power):
    """ Initialize each process's global variable power. """
    global power
    power = the_power

def raise_to_power(number):
   number = number ** power
   return number


if __name__ == '__main__':
   start_time = time.time()
   threads = 8
   power = 10
   FROM = 1
   TO = 1000000
   cnt = TO - FROM
   # changed to a generator expression to save memory:
   #numbers = (x for x in range(FROM, TO))
   # or just:
   numbers = range(FROM, TO)
   chunksize, rem = divmod(cnt, threads)
   if rem: chunksize += 1
   # initialize each pool process with power:
   with Pool(threads, initializer=init_pool, initargs=(power,)) as p:
      Product = p.map(raise_to_power,numbers,chunksize)
   print(time.time() - start_time)

Further Explanation

When a sub-process is created on a platform that uses OS call spawn to create the new process rather than fork, the new process does not inherit a copy of whatever global variables had been set in the parent process. Instead execution starts at the top of the program. But because you have if __name__ == '__main__': protecting the code that both creates the new processes and initializes power (which is required to avoid a recursive loop re-creating new processes ad infinitum), the statement power = 10 never gets executed in the sub-processes and voilà . When you move power = 10 outside of if __name__ == '__main__': , then it will get executed even when the sub-processes are created.