单生产者多消费者

Question

我希望在执行多线程编程时在 Python 中拥有一个生产者、多个消费者架构。 我希望有这样的操作：

1. 生产者产生数据
2. 消费者 1..N（N 是预先确定的）等待数据到达（阻塞），然后以不同的方式处理 SAME 数据。

所以我需要所有的消费者从生产者那里获得相同的数据。

当我使用 Queue 执行此操作时，我意识到除了第一个使用者之外的所有人都会对我拥有的实现感到饥饿。

一种可能的解决方案是为每个消费者线程设置一个唯一的队列，其中相同的数据由生产者推送到多个队列中。 有一个更好的方法吗？

from threading import Thread
import time
import random
from Queue import Queue

my_queue = Queue(0)

def Producer():
    global my_queue
    my_list = []
    for each in range (50):
        my_list.append(each)
    my_queue.put(my_list)

def Consumer1():
    print "Consumer1"
    global my_queue
    print my_queue.get()
    my_queue.task_done()

def Consumer2():
    print "Consumer2"
    global my_queue
    print my_queue.get()
    my_queue.task_done()


P = Thread(name = "Producer", target = Producer)

C1 = Thread(name = "Consumer1", target = Consumer1)

C2 = Thread(name = "Consumer2", target = Consumer2)


P.start()

C1.start()

C2.start()

在上面的例子中，C2 被无限期阻塞，因为 C1 消耗 P1 产生的数据。 我更希望 C1 和 C2 都能够访问 P1 生成的相同数据。

感谢您提供任何代码/指针！

Answer 1

您的生产者只创建一项工作要做：

my_queue.put(my_list)

例如，将 my_list 放入两次，两个消费者都工作：

def Producer():
    global my_queue
    my_list = []
    for each in range (50):
        my_list.append(each)
    my_queue.put(my_list)
    my_queue.put(my_list)

因此，通过这种方式，您可以将两个作业放入具有相同列表的队列中。

但是我必须警告您：在没有线程同步的情况下修改不同线程中的相同数据通常是个坏主意。

无论如何，使用一个队列的方法对您不起作用，因为应该使用具有相同算法的线程处理一个队列。

因此，我建议您继续为每个消费者设置唯一的队列，因为其他解决方案并非那么简单。

Answer 2

那么每个线程队列怎么样？

作为启动每个消费者的一部分，您还将创建另一个队列，并将其添加到“所有线程队列”列表中。 然后启动生产者，将所有队列的列表传递给它，然后他可以将数据推送到所有队列中。

Answer 3

一个单一生产者和五个消费者的例子，验证。

from multiprocessing import Process, JoinableQueue
import time
import os

q = JoinableQueue()

def producer():
    for item in range(30):
        time.sleep(2)
        q.put(item)
    pid = os.getpid()
    print(f'producer {pid} done')


def worker():
    while True:
        item = q.get()
        pid = os.getpid()
        print(f'pid {pid} Working on {item}')
        print(f'pid {pid} Finished {item}')
        q.task_done()

for i in range(5):
    p = Process(target=worker, daemon=True).start()

producers = []
# it is easy to extend it to multi producers.
for i in range(1):
    p = Process(target=producer)
    producers.append(p)
    p.start()

# make sure producers done
for p in producers:
    p.join()

# block until all workers are done
q.join()
print('All work completed')

解释：

1. 在这个例子中，一个生产者和五个消费者。
2. JoinableQueue 用于确保存储在队列中的所有元素都将被处理。 'task_done' 是让工作人员通知元素已完成。 'q.join()' 将等待所有标记为完成的元素。
3. 使用 #2，无需为每个工人加入等待。
4. 但是加入等待生产者将元素存储到队列中很重要。 否则，程序立即退出。

Answer 4

我知道这可能有点过头了，但是......使用 Qt 的signal/slot框架怎么样？ 为了一致性，可以使用QThread代替threading.Thread

from __future__ import annotations # Needed for forward Consumer typehint in register_consumer

from queue import Queue
from typing import List

from PySide2.QtCore import QThread, QObject, QCoreApplication, Signal, Slot, Qt
import time
import random

def thread_name():
    # Convenient class
    return QThread.currentThread().objectName()

class Producer(QThread):
    product_available = Signal(list)

    def __init__(self):
        QThread.__init__(self, objectName='ThreadProducer')
        self.consumers: List[Consumer] = list()
        # See Consumer class comments for info (exactly the same reason here)
        self.internal_consumer_queue = Queue()
        self.active = True

    def run(self):
        my_list = [each for each in range(5)]
        self.product_available.emit(my_list)
        print(f'Producer: from thread {QThread.currentThread().objectName()} I\'ve sent my products\n')
        while self.active:
            consumer: Consumer = self.internal_consumer_queue.get(block=True)
            print(f'Producer: {consumer} has told me it has completed his task with my product! '
                  f'(Thread {thread_name()})')
            if not consumer in self.consumers:
                raise ValueError(f'Consumer {consumer} was not registered')
            self.consumers.remove(consumer)
            if len(self.consumers) == 0:
                print('All consumers have completed their task! I\'m terminating myself')
                self.active = False

    @Slot(object)
    def on_task_done_by_consumer(self, consumer: Consumer):
        self.internal_consumer_queue.put(consumer)


    def register_consumer(self, consumer: Consumer):
        if consumer in self.consumers:
            return
        self.consumers.append(consumer)
        consumer.task_done_with_product.connect(self.on_task_done_by_consumer)


class Consumer(QThread):
    task_done_with_product = Signal(object)
    def __init__(self, name: str, producer: Producer):
        self.name = name
        # Super init and set Thread name
        QThread.__init__(self, objectName=f'Thread_Of_{self.name}')
        self.producer = producer
        # See method on_product_available doc
        self.internal_queue = Queue()

    def run(self) -> None:
        self.producer.product_available.connect(self.on_product_available, Qt.ConnectionType.UniqueConnection)
        # Thread loop waiting for product availability
        product = self.internal_queue.get(block=True)
        print(f'{self.name}: Product {product} received and elaborated in thread {thread_name()}\n\n')

        # Tell the producer I've done
        self.task_done_with_product.emit(self)

        # Now the thread is naturally closed

    @Slot(list)
    def on_product_available(self, product: list):
        """
        As a limitation of PySide, it seems that list are not supported for QueuedConnection. This work around using
        internal queue might solve
        """
        # This is executed in Main Loop!
        print(f'{self.name}: In thread {thread_name()} I received the product, and I\'m queuing it for being elaborated'
              f'in consumer thread')
        self.internal_queue.put(product)
        # Quit the thread
        self.active = False

    def __repr__(self):
        # Needed in case of exception for representing current consumer
        return f'{self.name}'


# Needed to executed main and threads event loops
app = QCoreApplication()

QThread.currentThread().setObjectName('MainThread')

producer = Producer()

c1 = Consumer('Consumer1', producer)
c1.start()
producer.register_consumer(c1)

c2 = Consumer('Consumer2', producer)
c2.start()
producer.register_consumer(c2)

producer.product_available.connect(c1.on_product_available)
producer.product_available.connect(c2.on_product_available)

# Start Producer thread for LAST!
producer.start()

app.exec_()

结果：

Producer: from thread ThreadProducer I've sent my products
Consumer1: In thread MainThread I received the product, and I'm queuing it for being elaboratedin consumer thread
Consumer1: Product [0, 1, 2, 3, 4] received and elaborated in thread Thread_Of_Consumer1
Consumer2: In thread MainThread I received the product, and I'm queuing it for being elaboratedin consumer thread
Consumer2: Product [0, 1, 2, 3, 4] received and elaborated in thread Thread_Of_Consumer2
Producer: Consumer1 has told me it has completed his task with my product! (Thread ThreadProducer)
Producer: Consumer2 has told me it has completed his task with my product! (Thread ThreadProducer)
All consumers have completed their task! I'm terminating myself

笔记：

• 分步说明在代码注释中。 如果有什么不清楚的地方，我会尽力澄清
• 不幸的是，我还没有找到一种使用QueueConnection （此处为文档）的方法，以便直接将 Slot 执行到正确的线程中：内部队列已用于将信息从主循环传递到正确的线程（生产者和消费者）。 似乎list和object不能在 PySide/pyqt 中进行元注册以进行排队