[英]Checking for empty Queue in python's multiprocessing
I have a program using python's packages multiprocessing and Queue.我有一个使用 python 的包 multiprocessing 和 Queue 的程序。 One of my functions have this structure:
我的功能之一具有以下结构:
from multiprocessing import Process, Queue
def foo(queue):
while True:
try:
a = queue.get(block = False)
doAndPrintStuff(a)
except:
print "the end"
break
if __name__ == "__main__"
nthreads = 4
queue = Queue.Queue()
# put stuff in the queue here
for stuff in moreStuff:
queue.put(stuff)
procs = [Process(target = foo, args = (queue,)) for i in xrange(nthreads)]
for p in procs:
p.start()
for p in procs:
p.join()
the idea is that when I try to extract from the queue and it is empty, it'll raise an exception and terminate the loop.这个想法是,当我尝试从队列中提取并且它为空时,它会引发异常并终止循环。 So I have two questions:
所以我有两个问题:
1) is this a safe idiom? 1)这是一个安全的习语吗? Are there better ways to do this?
有没有更好的方法来做到这一点?
2) I tried to find what is the exact exception that is raised when I try to .get()
from an empty queue. 2)我试图找出当我尝试从空队列中
.get()
时引发的确切异常是什么。 Currently my program is catching all exceptions, which sucks when the error is somewhere else and I only get a "the end" message.目前我的程序正在捕获所有异常,当错误发生在其他地方时这很糟糕,我只收到“结束”消息。
I tried:我试过:
import Queue
queue = Queue.Queue()
[queue.put(x) for x in xrange(10)]
try:
print queue.get(block = False)
except Queue.Empty:
print "end"
break
but I got the error as if I hadn't caught the exception.但我收到了错误,就好像我没有发现异常一样。 What's the correct exception to catch?
捕获的正确异常是什么?
The exception should be Queue.Empty
. 例外应该是
Queue.Empty
。 But are you sure you got the same error? 但你确定你得到了同样的错误吗? In your second example, you also switched the queue itself from
multiprocessing.Queue
to Queue.Queue
, which I think may be the problem. 在第二个示例中,您还将队列本身从
multiprocessing.Queue
切换到Queue.Queue
,我认为这可能是问题所在。
It might seem strange, but you have to use the multiprocessing.Queue
class, but use the Queue.Empty
exception (which you have to import yourself from the Queue
module) 这可能看起来很奇怪,但您必须使用
multiprocessing.Queue
类,但使用Queue.Empty
异常(您必须自己从Queue
模块导入)
It appears that the Queue is empty until the put buffers are flushed, which may take a while. 在刷新put缓冲区之前,Queue似乎是空的,这可能需要一段时间。
The solution to our problem is to use sentinels , or maybe the built-in task_done() call: 我们的问题的解决方案是使用 哨兵 ,也许内置task_done()调用:
task_done()
task_done()
Indicate that a formerly enqueued task is complete.
表示以前排队的任务已完成。 Used by queue consumer threads.
由队列使用者线程使用。 For each get() used to fetch a task, a subsequent call to task_done() tells the queue that the processing on the task is complete.
对于用于获取任务的每个get(),对task_done()的后续调用会告知队列该任务的处理已完成。
If a join() is currently blocking, it will resume when all items have been processed (meaning that a task_done() call was received for every item that had been put() into the queue).
如果join()当前正在阻塞,则它将在所有项目都已处理后恢复(这意味着已为每个已放入队列的项目收到task_done()调用)。
Raises a ValueError if called more times than there were items placed in the queue.
如果调用的次数超过队列中放置的项目,则引发ValueError。
Here's an example- As @Steven said above, you need to use the queue.Empty exception from the standard queue.这是一个示例 - 正如@Steven 上面所说,您需要使用标准队列中的 queue.Empty 异常。 The note from documentation ( https://docs.python.org/3/library/multiprocessing.html ):
文档中的注释( https://docs.python.org/3/library/multiprocessing.html ):
Note
笔记
multiprocessing uses the usual queue.Empty and queue.Full exceptions to signal a timeout.
multiprocessing 使用通常的 queue.Empty 和 queue.Full 异常来表示超时。 They are not available in the multiprocessing namespace so you need to import them from queue.
它们在多处理命名空间中不可用,因此您需要从队列中导入它们。
basic example:基本示例:
from multiprocessing import Process, Queue, Manager
import queue
def firstPass(q):
driver = getDriver()
while True:
try:
link = q.get_nowait()
f(driver, link)
except queue.Empty:
logger.info("empty queue")
driver.close()
break
Try reading the queue library docs. 尝试阅读队列库文档。 Aren't you looking for Queue.empty() ?
你不是在寻找Queue.empty()吗?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.