生成过程中的 Static 变量未初始化

Question

It is strange for me, that "spawn" process does not copy static variables of class, but with "fork" everything is ok.

Code:

import typing as t
import multiprocessing as mp
from concurrent.futures import ProcessPoolExecutor


class Foo:
    static_var: int

    def __init__(self):
        pass


def func(foo_class: t.Type[Foo]):
    return foo_class.static_var


if __name__ == "__main__":
    # with ProcessPoolExecutor(mp_context=mp.get_context("fork")) as executor:
    with ProcessPoolExecutor(mp_context=mp.get_context("spawn")) as executor:
        foo_class = Foo
        foo_class.static_var = 10
        res = executor.map(func, [foo_class, foo_class])
        print(list(res))
    print('Done')

Output "fork":

[10, 10]
Done

Output "spawn":

AttributeError: type object 'Foo' has no attribute 'static_var'

Python version: 3.8.5

I have no idea how to overcome it with a spawn (and I do not totally understand why it does not work). Spawn process starts own interpreter and import modules again (and classes?), that is why static variables are not initialised? How it is possible to pass a variable via a class?

Answer 1

In multiprocessing, the spwan or fork-server worker processes are new python processes.
All memory states are fresh and the if __name__ == "__main__:" block is not executed. The static class variable in this sense is not static.

In the fork worker processes, the whole process memory is copied so the class variable is likely copied but any change in one process do not affect other processes.

Two possible solutions:

• Modify the task function and arguments such that the task is self-contained and do not depend on non-static environment.
• Use the initializer argument in ProcessPoolExecutor to set up the static variable correctly.

def worker_initializer():
    foo_class = Foo
    foo_class.static_var = 10

with ProcessPoolExecutor(
    initializer=worker_initializer,
    mp_context=mp.get_context("spawn")
) as executor:
    pass