[英]Python multiprocessing and multiprocessing.Queue
I'm trying to implement a producer-consumer scenario using multiprocessing and a queue; 我正在尝试使用多处理和队列来实现生产者-消费者方案; the main process is the producer and two subprocesses consuming data from the queue.
主要流程是生产者和两个子流程,它们使用队列中的数据。 This works while nothing unusual happens, but the twist is that I want to be able to restart workers in case they die (
kill -9 workerpid
). 这可以正常工作 ,但没有发生任何异常情况,但有一个转折是,我希望能够重新启动工作程序以防他们死亡(
kill -9 workerpid
)。 However, when I kill one or both workers, they start saying "queue was empty" even if the main process keeps stuffing data in the queue. 但是,当我杀死一个或两个工人时,即使主进程将数据填充到队列中,他们也开始说“队列为空”。
What am I missing here? 我在这里想念什么? (using Python 2.7.3 on Ubuntu 12.04)
(在Ubuntu 12.04上使用Python 2.7.3)
import sys
import time
import multiprocessing
from Queue import Empty
workers = []
fqueue = multiprocessing.Queue()
class Worker(multiprocessing.Process):
def run(self):
queue = self._args[0]
print "{0} starting up, queue at: {1}".format(self.pid, queue)
while True:
try:
obj = queue.get(block=True, timeout=1)
print "{0}: got from queue: {1}".format(self.pid, obj)
except Empty:
print "{0}: queue was empty".format(self.pid)
continue
except IOError, e:
print "{0}: got IOError on queue: {1}".format(self.pid, e)
return
if __name__ == "__main__":
print "zipper starting up (2 workers)"
for _ in range(0, 2):
p = Worker(args=(fqueue,))
workers.append(p)
p.start()
cnt = 0
while True:
for i in range(0, len(workers)):
p = workers[i]
if not p.is_alive():
print "main: worker {0} is not alive".format(p.pid)
p = Worker(args=(fqueue,))
print "main: restarted worker: {0}".format(p)
p.start()
workers[i] = p
print "main: tick"
cnt += 1
fqueue.put(cnt)
time.sleep(2)
Have you seen the warning in the documentation ?: 您在文档中看到警告了吗?:
Warning
警告
If a process is killed using Process.terminate() or os.kill() while it is trying to use a Queue, then the data in the queue is likely to become corrupted.
如果在尝试使用队列时使用Process.terminate()或os.kill()将其杀死,则队列中的数据可能会损坏。 This may cause any other process to get an exception when it tries to use the queue later on.
这可能会导致其他进程稍后尝试使用队列时获得异常。
So killing a process that's using a queue can make the whole queue unusable. 因此,杀死正在使用队列的进程可能会使整个队列无法使用。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.