[英]How to terminate Producer-Consumer threads from main thread in Python?
I have a Producer and a Consumer thread ( threading.Thread
), which share a queue
of type Queue
. 我有一个生产者和消费者螺纹(
threading.Thread
),它们共享一个queue
类型的Queue
。
Producer run
: 制片人
run
:
while self.running:
product = produced() ### I/O operations
queue.put(product)
Consumer run
: 消费者
run
:
while self.running or not queue.empty():
product = queue.get()
time.sleep(several_seconds) ###
consume(product)
Now I need to terminate both threads from main thread, with the requirement that queue
must be empty (all consumed) before terminating. 现在我需要终止主线程中的两个线程,并要求
queue
在终止之前必须为空(全部已消耗)。
Currently I'm using code like below to terminate these two threads: 目前我正在使用如下代码来终止这两个线程:
main thread stop
: 主线程
stop
:
producer.running = False
producer.join()
consumer.running = False
consumer.join()
But I guess it's unsafe if there are more consumers. 但我想如果有更多的消费者,这是不安全的。
In addition, I'm not sure whether the sleep
will release schedule to the producer so that it can produce more products. 另外,我不确定
sleep
是否会向生产者发布时间表,以便它可以生产更多产品。 In fact, I find the producer keeps "starving" but I'm not sure whether this is the root cause. 事实上,我发现生产者一直“挨饿”,但我不确定这是否是根本原因。
Is there a decent way to deal with this case? 有没有一个体面的方法来处理这个案子?
You can put a sentinel object in queue to signal end of tasks, causing all consumers to terminate: 您可以将一个Sentinel对象放入队列以指示任务结束,从而导致所有使用者终止:
_sentinel = object()
def producer(queue):
while running:
# produce some data
queue.put(data)
queue.put(_sentinel)
def consumer(queue):
while True:
data = queue.get()
if data is _sentinel:
# put it back so that other consumers see it
queue.put(_sentinel)
break
# Process data
This snippet is shamelessly copied from Python Cookbook 12.3. 这个片段是从Python Cookbook 12.3中无耻地复制的。
_sentinel
to mark end of queue. _sentinel
标记队列结束。 None
also works if no task produced by producer is None
, but using a _sentinel
is safer for the more general case. None
还工作,如果没有由生产者生产的任务是None
,但使用_sentinel
是更普遍的情况下,更安全。 Edit 2: 编辑2:
a) The reason your consumers keep taking so much time is because your loop runs continously even when you have no data. a)消费者花费这么多时间的原因是因为即使你没有数据,你的循环也会持续运行。
b) I added code at that bottom that shows how to handle this. b)我在底部添加了代码,展示了如何处理这个问题。
If I understood you correctly, the producer/consumer is a continuous process, eg it is acceptable to delay the shutdown until you exit the current blocking I/O and process the data you received from that. 如果我理解正确,生产者/消费者是一个连续的过程,例如,可以延迟关闭,直到您退出当前的阻塞I / O并处理从中收到的数据。
In that case, to shut down your producer and consumer in an orderly fashion, I would add communication from the main thread to the producer thread to invoke a shutdown. 在这种情况下,要以有序的方式关闭您的生产者和消费者,我会添加从主线程到生产者线程的通信以调用关闭。 In the most general case, this could be a queue that the main thread can use to queue a "shutdown" code, but in the simple case of a single producer that is to be stopped and never restarted, it could simply be a global shutdown flag.
在最一般的情况下,这可能是主线程可用于排队“关闭”代码的队列,但在单个生产者的简单情况下,要停止并且从不重新启动,它可能只是全局关闭旗。
Your producer should check this shutdown condition (queue or flag) in its main loop right before it would start a blocking I/O operation (eg after you have finished sending other data to the consumer queue). 您的生产者应该在它开始阻塞I / O操作之前检查其主循环中的此关闭条件(队列或标志)(例如,在您将其他数据发送到使用者队列之后)。 If the flag is set, then it should put a special end-of-data code (that does not look like your normal data) on the queue to tell the consumer that a shut down is occurring, and then the producer should return (terminate itself).
如果设置了标志,那么它应该在队列上放置一个特殊的数据结尾代码(看起来不像普通数据),告诉消费者正在关闭,然后生产者应该返回(终止本身)。
The consumer should be modified to check for this end-of-data code whenever it pulls data out of the queue. 应该修改使用者,以便在将数据从队列中拉出时检查此数据结束代码。 If the end-of-data code is found, it should do an orderly shutdown and return (terminating itself).
如果找到数据结束代码,它应该按顺序关闭并返回(终止自身)。
If there are multiple consumers, then the producer could queue multiple end-of-data messages -- one for each consumer -- before it shuts down. 如果有多个消费者,那么生产者可以在关闭之前为多个数据结束消息排队 - 每个消费者一个消息。 Since the consumers stop consuming after they read the message, they will all eventually shut down.
由于消费者在阅读消息后停止消费,他们最终都将关闭。
Alternatively, if you do not know up-front how many consumers there are, then part of the orderly shut down of the consumer could be re-queueing the end-of-data code. 或者,如果您事先不知道有多少消费者,那么消费者有序关闭的部分可能会重新排队数据结束代码。
This will insure that all consumers eventually see the end-of-data code and shut down, and when all are done, there will be one remaining item in the queue -- the end-of-data code queued by the last consumer. 这将确保所有消费者最终看到数据结束代码并关闭,并且当完成所有操作后,队列中将剩余一个项目 - 由最后一个消费者排队的数据结束代码。
EDIT: 编辑:
The correct way to represent your end-of-data code is highly application dependent, but in many cases a simple None
works very well. 表示数据结束代码的正确方法取决于应用程序,但在许多情况下,简单的
None
非常有效。 Since None
is a singleton, the consumer can use the very efficient if data is None
construct to deal with the end case. 由于
None
是单例,因此if data is None
构造来处理最终案例,则消费者可以使用非常有效的方法。
Another possibility that can be even more efficient in some cases is to set up a try /except
outside your main consumer loop, in such a way that if the except happened, it was because you were trying to unpack the data in a way that always works except for when you are processing the end-of-data code. 在某些情况下,另一种可能更有效的可能性是在主消费者循环之外设置一个
try /except
,这样如果发生了除外,那是因为你试图以一种总是以一种方式解压缩数据除了处理数据结束代码时,它们可以正常工作。
EDIT 2: 编辑2:
Combining these concepts with your initial code, now the producer does this: 将这些概念与您的初始代码相结合,现在生产者就是这样做的:
while self.running:
product = produced() ### I/O operations
queue.put(product)
for x in range(number_of_consumers):
queue.put(None) # Termination code
Each consumer does this: 每个消费者这样做:
while 1:
product = queue.get()
if product is None:
break
consume(product)
The main program can then just do this: 主程序可以这样做:
producer.running = False
producer.join()
for consumer in consumers:
consumer.join()
One observation from your code is that, your consumer
will keep on looking for getting some thing from the queue, ideally you should handle that by keeping some timeout
and handle Empty
exception for the same like below, ideally this helps to check the while self.running or not queue.empty()
for every timeout
. 您的代码中的一个观察是,您的
consumer
将继续寻找从队列中获取某些东西,理想情况下您应该通过保持一些timeout
来处理它并处理相同的Empty
异常,如下所示,理想情况下这有助于检查while self.running or not queue.empty()
每次timeout
都while self.running or not queue.empty()
。
while self.running or not queue.empty():
try:
product = queue.get(timeout=1)
except Empty:
pass
time.sleep(several_seconds) ###
consume(product)
I did simulated your situation and created producer
and consumer
threads, Below is the sample code that is running with 2 producers
and 4 consumers
it's working very well. 我确实模拟了你的情况并创建了
producer
和consumer
线程。下面是与2个producers
和4个consumers
一起运行的示例代码,它运行良好。 hope this helps you! 希望这对你有所帮助!
import time
import threading
from Queue import Queue, Empty
"""A multi-producer, multi-consumer queue."""
# A thread that produces data
class Producer(threading.Thread):
def __init__(self, group=None, target=None, name=None,
args=(), kwargs=None, verbose=None):
threading.Thread.__init__(self, group=group, target=target, name=name,
verbose=verbose)
self.running = True
self.name = name
self.args = args
self.kwargs = kwargs
def run(self):
out_q = self.kwargs.get('queue')
while self.running:
# Adding some integer
out_q.put(10)
# Kepping this thread in sleep not to do many iterations
time.sleep(0.1)
print 'producer {name} terminated\n'.format(name=self.name)
# A thread that consumes data
class Consumer(threading.Thread):
def __init__(self, group=None, target=None, name=None,
args=(), kwargs=None, verbose=None):
threading.Thread.__init__(self, group=group, target=target, name=name,
verbose=verbose)
self.args = args
self.kwargs = kwargs
self.producer_alive = True
self.name = name
def run(self):
in_q = self.kwargs.get('queue')
# Consumer should die one queue is producer si dead and queue is empty.
while self.producer_alive or not in_q.empty():
try:
data = in_q.get(timeout=1)
except Empty, e:
pass
# This part you can do anything to consume time
if isinstance(data, int):
# just doing some work, infact you can make this one sleep
for i in xrange(data + 10**6):
pass
else:
pass
print 'Consumer {name} terminated (Is producer alive={pstatus}, Is Queue empty={qstatus})!\n'.format(
name=self.name, pstatus=self.producer_alive, qstatus=in_q.empty())
# Create the shared queue and launch both thread pools
q = Queue()
producer_pool, consumer_pool = [], []
for i in range(1, 3):
producer_worker = Producer(kwargs={'queue': q}, name=str(i))
producer_pool.append(producer_worker)
producer_worker.start()
for i in xrange(1, 5):
consumer_worker = Consumer(kwargs={'queue': q}, name=str(i))
consumer_pool.append(consumer_worker)
consumer_worker.start()
while 1:
control_process = raw_input('> Y/N: ')
if control_process == 'Y':
for producer in producer_pool:
producer.running = False
# Joining this to make sure all the producers die
producer.join()
for consumer in consumer_pool:
# Ideally consumer should stop once producers die
consumer.producer_alive = False
break
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.