多处理池生成初始化非常慢，16 个进程为 100 秒

Question

In my program I need to spawn 16 multiprocessing pools, to utilise 16 cores on my 3950x. I have an initialiser which initialises three global variables in the spawn child processes (totalling around 300kb).

Before the using the initialiser each process took about 1s to spawn. Now 16 process takes around 100s in total? Any idea why this is now so slow: Sample code below:

def set_mp_global_vars(bc, bm, bf_n):
    """mp initialiser which sets global vars to reduce mp overhead"""
    global base_comps, base_matches, bf_names
    base_comps = bc
    base_matches = bm
    bf_names = bf_n

int_pool_workers = mp.cpu_count()
pool = mp.Pool(processes=int_pool_workers, initializer=set_mp_global_vars,
               initargs=(base_comps, base_matches, bf_names))

Answer 1

Using Python 3.9.2 on Windows 10 and spawning 30 processes on Threadripper CPU take 23s with Pycharm debugging on. If I run without debugging, it takes 0.21s.

pydev debugger slows down mp.Pool by a factor of 100x.

You may also look at: python-3-6-multiprocessing-pool-very-slow-to-start-using-windows