How to avoid loading a parent module in a forked process with Pythons multiprocessing

Question

When you create a Pool of processes using Python's multiprocessing , those processes will fork and globals in the parent process will show up in the child processes, noted in this question below:

How can I restrict the scope of a multiprocessing process?

This appears to include imported modules. That's a problem for modules that have side effects in __init__ . Tensorflow is one such module, as soon as it's imported it tries to allocate memory to the GPU. This causes the sub process to crash because the parent already took that action.

Is there a good way to avoid the tensorflow module loading in the forked process?

Python 2.7 on Ubuntu (posix)

Answer 1

After much debugging I realize my problem statement wasn't really sufficient. The problem was that I did load tensorflow in one of the sub processes (I forgot!) and the sub processes needed to use the CPU only, not the GPU. I was forced to change environment variables to disable CUDA on the subprocesses with:

os.environ['CUDA_VISIBLE_DEVICES'] = ''
p = multiprocessing.Pool(processes=4)
os.environ['CUDA_VISIBLE_DEVICES'] = '0'

Answer 2

除了David Parks的自我回答之外，在导入TensorFlow或可以被os.fork()破坏的模块之前创建Pool和worker实例更安全。