Using the Manager (for Pool) in the function for multiprocessing (Windows 10)

Question

I am learning about Pool, Manager etc from multiprocessing. I want to use the namespace in Manager in my function. I took some code off the internet that highlights the problem of multiprocessing Manager in Windows. Here it is:

"""How to share data in multiprocessing with Manager.Namespace()"""
from multiprocessing import Pool, Manager

import numpy as np


# Create manager object in module-level namespace
mgr = Manager()
# Then create a container of things that you want to share to
# processes as Manager.Namespace() object.
config = mgr.Namespace()
# The Namespace object can take various data type
config.a = 1
config.b = '2'
config.c = [1, 2, 3, 4]


def func(i):
    """This is a function that we want our processes to call."""
    # You can modify the Namespace object from anywhere.
    config.z = i
    print('config is', config)
    # And they will still be shared (i.e. same id).
    print('id(config) = {:d}'.format(id(config)))


# This main func
def main():
    """The main function contain multiprocess.Pool codes."""
    # You can add to the Namespace object too.
    config.d = 10
    config.a = 5.25e6
    pool = Pool(1)
    pool.map(func, (range(20, 25)))
    pool.close()
    pool.join()


if __name__ == "__main__":
    # Let's print the config
    print(config)
    # Now executing main()
    main()
    # Again, you can add or modify the Namesapce object from anywhere.
    config.e = np.round(np.random.rand(2,2), 2)
    config.f = range(-3, 3)
    print(config)

The error is the following:

An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

This probably means that you are not using fork to start your
child processes and you have forgotten to use the proper idiom
in the main module:

    if __name__ == '__main__':
        freeze_support()
        ...

The "freeze_support()" line can be omitted if the program
is not going to be frozen to produce an executable.

The problem, I think, is that the manager is plopped in with a global variable. You cannot do this with Windows. As you can see, I am guarding the main, but that is not enough. What needs to be done is pass the manager to the function (probably into the map variables) somehow, but I don't know how to do this.

Answer 1

Yes, it looks like creating Manager as a global causes problems on Windows. Move it to the module main and pass the Namespace as a parameter. Pool.map() allows passing only one argument to a worker, so put multiple arguments, including the namespace, into a list. Pass a list of argument lists to Pool.map().

I may be wrong but I don't think you should expect/require the object ids not to change.

from multiprocessing import Pool, Manager

import numpy as np


def func(a):
    """This is a function that we want our processes to call."""
    (config, i) = a
    # You can modify the Namespace object from anywhere.
    config.z = i
    print('config is', config)
    # And they will still be shared (i.e. same id).
    print('id(config) = {:d}'.format(id(config)))


# This main func
def main(config):
    """The main function contain multiprocess.Pool codes."""
    # You can add to the Namespace object too.
    config.d = 10
    config.a = 5.25e6
    pool = Pool(1)
    pool.map(func, list([config, i] for i in range(20,25)))
    pool.close()
    pool.join()


if __name__ == "__main__":
    # Create manager object in module-level namespace
    mgr = Manager()
    # Then create a container of things that you want to share to
    # processes as Manager.Namespace() object.
    config = mgr.Namespace()
    # The Namespace object can take various data type
    config.a = 1
    config.b = '2'
    config.c = [1, 2, 3, 4]

    # Let's print the config
    print(config)
    # Now executing main()
    main(config)
    # Again, you can add or modify the Namesapce object from anywhere.
    config.e = np.round(np.random.rand(2,2), 2)
    config.f = range(-3, 3)
    print(config)