简体   繁体   English

python 3.6 multiprocessing.Pool()开始使用Windows的速度非常慢

[英]python 3.6 multiprocessing.Pool() very slow to start using Windows

Running Python 3.6.4 (64 bit) on Windows 10, AMD Threadripper 16 core CPU, 64 GB RAM, fast SSD. 在Windows 10,AMD Threadripper 16核心CPU,64 GB RAM,快速SSD上运行Python 3.6.4(64位)。 Nothing else is running (<2% CPU) or using RAM (55GB Free) before I start this test. 在开始此测试之前,没有其他东西在运行(<2%CPU)或使用RAM(55GB可用空间)。 Everything runs fast except calling mp.Pool() to setup the worker processes. 除了调用mp.Pool()设置工作进程外,其他所有程序都运行很快。 Once setup, running pool.map() is fast as expected. 设置完成后,运行pool.map()的速度将达到预期的速度。 Notes: Running local and tested with and w/o virtual env. 注意:在本地运行并通过虚拟环境进行测试。

Any fixes, workarounds, ideas or explanations would be appreciated. 任何修复,解决方法,想法或解释将不胜感激。 Thanks. 谢谢。

import multiprocessing as mp
import time

for x in range(2,15):
    t0 = time.perf_counter()
    with mp.Pool(processes=x) as pool:
        pass
    print('Done {0} processes in {1:.2f}s'.format(x, time.perf_counter() - t0))

Done 2 processes in 0.79s
Done 3 processes in 1.34s
Done 4 processes in 2.18s
...
Done 12 processes in 6.44s
Done 13 processes in 5.45s
Done 14 processes in 5.73s

After several hours, I determined it was a Bitdefender setting that was causing the problem. 几个小时后,我确定是导致问题的原因是Bitdefender设置。 Under antivirus there is a setting called, "Scan only new and changed files." 在防病毒下,有一个名为“仅扫描新文件和更改的文件”的设置。 If set to Off, the performance problem happens. 如果设置为“关”,则会发生性能问题。 Turn it on and the timings go down to 0.11s per 5 processes. 开启它,计时降为每5个进程0.11s。 For documentation purposes, I am running Bitdefender AV Plus 2018 (up to date as of today). 出于文档目的,我正在运行Bitdefender AV Plus 2018(到目前为止是最新的)。 I reported the problem to Bitdefender and it was escalated to the next tier of support. 我将此问题报告给Bitdefender,并将其升级为下一个支持层。

Thanks everyone for the input. 感谢大家的投入。

声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM