[英]Python's concurrent.futures hangs in both multithreading and multiprocessing on some systems
The code I wrote is to do some data analysis work, and it has been working well for several months now.我写的代码是做一些数据分析的工作,现在已经运行好几个月了。 But the size of my source data increased significantly recently, and I see that the code hangs now without errors at around the same point in execution (but not always at the same point).
但是最近我的源数据的大小显着增加,我看到代码现在在执行的同一点(但并不总是在同一点)挂起而没有错误。
The code looks like this:代码如下所示:
def submission_loop(data, submission):
# No loops in this function
# Do some data analysis
return result
def data_loop(arg1, arg2, data_row):
# Check this marker against all the criteria
results = []
for data in data_row:
for submission in submissions:
results.append(submission_loop(data, submission))
# Do something with result here
return results
if __name__ == '__main__':
with concurrent.futures.ProcessPoolExecutor(max_workers=cpu_count()) as executor:
chunksize = max(1, int(len(data_rows)/cpu_count()))
results = executor.map(functools.partial(data_loop, *args), data_rows, chunksize=chunksize)
results = list(results)
Note the three levels of loops.请注意三个级别的循环。
Now, I've tested this on 3 machines:现在,我已经在 3 台机器上进行了测试:
python:3.8.5
running on an Ubuntu 20 host.python:3.8.5
在 Ubuntu 20 主机上运行。python:3.8.5
running on a Windows 10 host using Docker Desktop.python:3.8.5
在 Windows 10 主机上运行,使用 ZC5FD214CDD3B2B3B42Z727E 桌面。python 3.8.5
.python 3.8.5
。 On both 1 and 2, the above mentioned issue is seen consistently.在 1 和 2 上,上述问题始终如一。 On 3, the task completes successfully.
3日,任务成功完成。
I changed this to use ThreadPoolExecutor
and the issue does not resolve, which makes me say that the number of cores is irrelevant here.我将其更改为使用
ThreadPoolExecutor
并且问题没有解决,这让我说核心数量在这里无关紧要。 If I remove concurrent.futures
usage and use a serial loop, it works perfectly.如果我删除
concurrent.futures
使用并使用串行循环,它会完美运行。
Is this a bug with concurrent.futures
?这是
concurrent.futures
的错误吗?
Further to my previous comment.继我之前的评论之后。 I noticed that
pebble
also was causing similar issues.我注意到
pebble
也引起了类似的问题。 I switched to futureproof
and the issue has been resolved.我切换到
futureproof
,问题已经解决。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.