Python 多处理和 sys.argv

Question

Are sys.argv values passed to the branches of multiprocessing? What is the correct way of passing argv to all branches of the multiprocess?

Let's suppose I have two files: test1.py:

import sys
if len(sys.argv) > 1:
    env = sys.argv[1]
else:
    env = 'test'

And main_code.py:

from test1 import *
import concurrent.futures


def f():
    if env == 'test':
        print('bu')
    else:
        print('not bu')

if __name__ == '__main__':
    with concurrent.futures.ProcessPoolExecutor(max_workers=2) as executor:
        for i in range(2):
            executor.submit(f)

I invoke from cmd main.code.py: python main_code.py zzz . Is the sys.argv[1] variable (which is 'zzz') passed on each invocation of executor.submit(f) as it was first obtained from import of text1.py? My confusion comes from the fact that concurrent.futures basically creates separate instances of threads of code by re-importing all the files.

Answer 1

On Windows, the spawn context is the only way to create worker processes.

1. sys.argv are copied to worker processes once.

2. Not all files are re-imported. Only the modules are required to unpickle the task function and arguments are imported.

3. In the worker, the original __main__ is actually called __mp_main__ . After copying sys.argv , the worker import __mp_main__ , which import test , so env is set correctly.

4. Though multiprocessing try to keep the environment similar, the worker process entry point is somewhere inside multiprocessing.spawn . Several items are mentioned there: sys.argv , sys.path , os.getcwd() . See get_preparation_data() and prepare() for details.

5. It can be verified with Task Manager or ps command that the worker process is started with different arguments.

I wrote a simple script called mp.py to print the arguments by running python3 mp.py hello world .

Output:

29836 process ['C:/xxxx/stackoverflow/mp.py'] <module '__main__' from 'C:/xxxx/stackoverflow/mp.py'>
29836 my name is main
29836 true main <module '__main__' from 'C:/xxxx/stackoverflow/mp.py'>
18464 process ['C:\\xxxx\\stackoverflow\\mp.py'] <module '__main__' (built-in)>
18464 worker <module '__mp_main__' from 'C:\\xxxx\\stackoverflow\\mp.py'>

mp.py:

from __future__ import annotations

import multiprocessing
import os
import sys
import time
from concurrent.futures import ProcessPoolExecutor


def list_modules(who_am_i):
    the_main = sys.modules.get('__main__')
    print(os.getpid(), who_am_i, the_main)


def main():
    list_modules('true main')
    mp_context = multiprocessing.get_context('spawn')
    # mp_context = multiprocessing.get_context('fork')
    # mp_context = multiprocessing.get_context('forkserver')
    with ProcessPoolExecutor(1, mp_context=mp_context) as executor:
        executor.submit(list_modules, 'worker').result()

        time.sleep(100)


# This message is print when this module is loaded. (none in fork, once in forkserver, multiple times in spawn)
print(os.getpid(), "process", sys.argv, sys.modules.get('__main__'))

if __name__ == '__main__':
    # Print once in the main process
    print(os.getpid(), "my name is main")
    main()