[英]futures.wait() or futures.as_completed() blocked while all futures are Completed or Cancelled
I have a bug with futures.as_completed() or futures.wait() that will be blocked indefinitely when all Futures are completed or cancelled.我有一个关于 futures.as_completed() 或 futures.wait() 的错误,当所有 Futures 完成或取消时,它将被无限期地阻止。
Here the steps to reproduce:这里重现的步骤:
After having submited Futures with ThreadPoolExecutor.submit()
, I'm waiting for my Futures with futures.as_completed()
or futures.wait()
.在使用
ThreadPoolExecutor.submit()
提交期货后,我正在等待使用futures.as_completed()
或futures.wait()
的期货。 In an other thread, I call ThreadPoolExecutor.shutdown()
with cancel_futures=True
and then in this same process, I'm waiting for Futures to complete with a timeout.在另一个线程中,我使用
cancel_futures=True
调用ThreadPoolExecutor.shutdown()
,然后在同一个过程中,我正在等待 Futures 超时完成。 The wait will return after the delay passed, with 2 lists: completed Futures and cancelled Futures.等待将在延迟过去后返回,有 2 个列表:已完成的期货和取消的期货。 There is no more pending Futures.
没有更多的未决期货。 However, the first
as_completed()
(or wait()
) in the main thread is still blocking.但是,主线程中的第一个
as_completed()
(或wait()
)仍然是阻塞的。
In the Python documenation , it is stated for return_when=ALL_COMPLETED
:在Python 文档中,说明
return_when=ALL_COMPLETED
:
The function will return when all futures finish or are cancelled .
当所有期货完成或被取消时,function 将返回。
And for as_completed()
对于
as_completed()
Returns [...] futures as they complete (finished or cancelled futures).
在完成时返回 [...] 期货(已完成或取消的期货)。
Which corresponds to my situation.这符合我的情况。 Is it a bug or am I missing something?
这是一个错误还是我错过了什么? I tried to call
shutdown()
in the same thread, it doesn't change anything.我试图在同一个线程中调用
shutdown()
,它没有改变任何东西。
Code sample:代码示例:
import signal
import time
from concurrent import futures
from concurrent.futures import Future, ALL_COMPLETED
from concurrent.futures import ThreadPoolExecutor
from typing import Dict, Set
class SubThreads:
def __init__(self):
self.running_futures_url: Dict[str, Future] = {}
self.webpage_crawler_th_pool = ThreadPoolExecutor(2)
def shutdown(self):
print("Waiting for lasts URL threads")
self.webpage_crawler_th_pool.shutdown(wait=False, cancel_futures=True)
finished_futures, still_running_futures = futures.wait(
self.running_futures_url.values(), return_when=ALL_COMPLETED, timeout=5,
)
print("Shutdown done, remaining threads", len(still_running_futures))
def crawl_url(self, url):
print("Crawling webpage", url)
time.sleep(3)
print("Webpage crawled", url)
return "URL Crawled"
def run(self):
urls = ['1', '2', '3', '4', '5']
for url in urls:
running_th = self.webpage_crawler_th_pool.submit(self.crawl_url, url)
self.running_futures_url[url] = running_th
print("Waiting for URLs to be crawled")
# for _future in futures.as_completed(self.running_futures_url.values()):
# print("Future result:", _future.result()) # Will only return and print first 2 started (and completed) Futures
finished_futures, still_running_futures = futures.wait(
self.running_futures_url.values(), return_when=ALL_COMPLETED
)
print("SubThread finished (never called)", finished_futures, still_running_futures)
sub_thread = SubThreads()
def signal_handler(sig, frame):
print("Signal caught, exiting ...", sig)
sub_thread.shutdown()
signal.signal(signal.SIGINT, signal_handler)
signal.signal(signal.SIGTERM, signal_handler)
sub_thread.run()
I would not use shutdown
with wait=False
.我不会将
shutdown
与wait=False
一起使用。
See the docs:请参阅文档:
shutdown(wait=True)
关机(等待=真)
Stops accepting new tasks.
停止接受新任务。 It waits for all the running tasks to complete if wait is True.
如果 wait 为 True,它会等待所有正在运行的任务完成。
So since you pass wait=False
, your as_completed
will wait forever, because the ThreadPoolExecutor
will never stop running until you call shutdown
with wait=True
.因此,由于您通过
wait=False
,您的as_completed
将永远等待,因为ThreadPoolExecutor
将永远不会停止运行,直到您使用wait=True
调用shutdown
。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.