导入模块的多处理不起作用

Question

I have a problem with multiprocessing when importing a module, which can be represented with the following example.

I have a module named tmp1.py with the following content:

from multiprocessing import Pool

def mul2(x):
    return [a*2 for a in x]

def calculate(func, x):
    r = []
    if __name__ == '__main__':
        p = Pool(2)
        r.append(p.map(func, x))
        p.close(), p.join()
    return r

I import it to another file tmp2.py with the following content:

import tmp1

inputs = [[1, 2, 3],
          [4, 5, 6],
          [7, 8, 9]]    

result = tmp1.calculate(tmp1.mul2, inputs)
print(result)

What I expect after running it, are values of the input multiplied by 2. However, after I run it all I get is an empty list. I work on Windows, python 3.10.

If I copy-paste all the content of tmp1.py to tmp2.py and adapt the function calls it works fine.

Where's the problem?

Answer 1

if __name__ == '__main__': is required to control code that creates new processes on platforms such as Windows that use the spawn method for creating new processes.

When the spawn method is being used the child process is created by creating an "empty" process, ie one that inherits nothing from the main process, into which a new Python interpreter is launched. This new interpreter must re-read all the source files already referenced in order to execute all statements at global scope in order to re-create global variables, function definitions, etc. in the new child process's address space before invoking, in your case, function tmp1.mul2 . In the new process __name__ will not be '__main__' and therefore we will not be re-executing recursively the code that created this new process to begin with.

The code that creates new processes at global scope , which is the scope we need to concern ourselves with, is actually in the main script, specifically result = tmp1.calculate(tmp1.mul2, inputs) . It is this code that needs to be within the if __name__ == '__main__': block:

File tmp2.py

if __name__ == '__main__':
    import tmp1

    inputs = [[1, 2, 3],
              [4, 5, 6],
              [7, 8, 9]]

    result = tmp1.calculate(tmp1.mul2, inputs)
    print(result)

I have put within the block all global statements that do not need to be executed within the new child process in order for it to have its storage initialized with whatever it might need for execution, ie all of the tmp2.py code.

Then tmp1.py becomes:

def mul2(x):
    return [a*2 for a in x]

def calculate(func, x):
    from multiprocessing import Pool

    r = []
    p = Pool(2)
    r.append(p.map(func, x))
    p.close()
    p.join()
    return r

The above prints:

[[[2, 4, 6], [8, 10, 12], [14, 16, 18]]]

This is a list of a single list. Perhaps, you really want:

def mul2(x):
    return [a*2 for a in x]

def calculate(func, x):
    from multiprocessing import Pool

    p = Pool(2)
    r = p.map(func, x)
    p.close()
    p.join()
    return r

The prints:

[[2, 4, 6], [8, 10, 12], [14, 16, 18]]