單生產者多消費者

Question

我希望在執行多線程編程時在 Python 中擁有一個生產者、多個消費者架構。 我希望有這樣的操作：

1. 生產者產生數據
2. 消費者 1..N（N 是預先確定的）等待數據到達（阻塞），然后以不同的方式處理 SAME 數據。

所以我需要所有的消費者從生產者那里獲得相同的數據。

當我使用 Queue 執行此操作時，我意識到除了第一個使用者之外的所有人都會對我擁有的實現感到飢餓。

一種可能的解決方案是為每個消費者線程設置一個唯一的隊列，其中相同的數據由生產者推送到多個隊列中。 有一個更好的方法嗎？

from threading import Thread
import time
import random
from Queue import Queue

my_queue = Queue(0)

def Producer():
    global my_queue
    my_list = []
    for each in range (50):
        my_list.append(each)
    my_queue.put(my_list)

def Consumer1():
    print "Consumer1"
    global my_queue
    print my_queue.get()
    my_queue.task_done()

def Consumer2():
    print "Consumer2"
    global my_queue
    print my_queue.get()
    my_queue.task_done()


P = Thread(name = "Producer", target = Producer)

C1 = Thread(name = "Consumer1", target = Consumer1)

C2 = Thread(name = "Consumer2", target = Consumer2)


P.start()

C1.start()

C2.start()

在上面的例子中，C2 被無限期阻塞，因為 C1 消耗 P1 產生的數據。 我更希望 C1 和 C2 都能夠訪問 P1 生成的相同數據。

感謝您提供任何代碼/指針！

Answer 1

您的生產者只創建一項工作要做：

my_queue.put(my_list)

例如，將 my_list 放入兩次，兩個消費者都工作：

def Producer():
    global my_queue
    my_list = []
    for each in range (50):
        my_list.append(each)
    my_queue.put(my_list)
    my_queue.put(my_list)

因此，通過這種方式，您可以將兩個作業放入具有相同列表的隊列中。

但是我必須警告您：在沒有線程同步的情況下修改不同線程中的相同數據通常是個壞主意。

無論如何，使用一個隊列的方法對您不起作用，因為應該使用具有相同算法的線程處理一個隊列。

因此，我建議您繼續為每個消費者設置唯一的隊列，因為其他解決方案並非那么簡單。

Answer 2

那么每個線程隊列怎么樣？

作為啟動每個消費者的一部分，您還將創建另一個隊列，並將其添加到“所有線程隊列”列表中。 然后啟動生產者，將所有隊列的列表傳遞給它，然后他可以將數據推送到所有隊列中。

Answer 3

一個單一生產者和五個消費者的例子，驗證。

from multiprocessing import Process, JoinableQueue
import time
import os

q = JoinableQueue()

def producer():
    for item in range(30):
        time.sleep(2)
        q.put(item)
    pid = os.getpid()
    print(f'producer {pid} done')


def worker():
    while True:
        item = q.get()
        pid = os.getpid()
        print(f'pid {pid} Working on {item}')
        print(f'pid {pid} Finished {item}')
        q.task_done()

for i in range(5):
    p = Process(target=worker, daemon=True).start()

producers = []
# it is easy to extend it to multi producers.
for i in range(1):
    p = Process(target=producer)
    producers.append(p)
    p.start()

# make sure producers done
for p in producers:
    p.join()

# block until all workers are done
q.join()
print('All work completed')

解釋：

1. 在這個例子中，一個生產者和五個消費者。
2. JoinableQueue 用於確保存儲在隊列中的所有元素都將被處理。 'task_done' 是讓工作人員通知元素已完成。 'q.join()' 將等待所有標記為完成的元素。
3. 使用 #2，無需為每個工人加入等待。
4. 但是加入等待生產者將元素存儲到隊列中很重要。 否則，程序立即退出。

Answer 4

我知道這可能有點過頭了，但是......使用 Qt 的signal/slot框架怎么樣？ 為了一致性，可以使用QThread代替threading.Thread

from __future__ import annotations # Needed for forward Consumer typehint in register_consumer

from queue import Queue
from typing import List

from PySide2.QtCore import QThread, QObject, QCoreApplication, Signal, Slot, Qt
import time
import random

def thread_name():
    # Convenient class
    return QThread.currentThread().objectName()

class Producer(QThread):
    product_available = Signal(list)

    def __init__(self):
        QThread.__init__(self, objectName='ThreadProducer')
        self.consumers: List[Consumer] = list()
        # See Consumer class comments for info (exactly the same reason here)
        self.internal_consumer_queue = Queue()
        self.active = True

    def run(self):
        my_list = [each for each in range(5)]
        self.product_available.emit(my_list)
        print(f'Producer: from thread {QThread.currentThread().objectName()} I\'ve sent my products\n')
        while self.active:
            consumer: Consumer = self.internal_consumer_queue.get(block=True)
            print(f'Producer: {consumer} has told me it has completed his task with my product! '
                  f'(Thread {thread_name()})')
            if not consumer in self.consumers:
                raise ValueError(f'Consumer {consumer} was not registered')
            self.consumers.remove(consumer)
            if len(self.consumers) == 0:
                print('All consumers have completed their task! I\'m terminating myself')
                self.active = False

    @Slot(object)
    def on_task_done_by_consumer(self, consumer: Consumer):
        self.internal_consumer_queue.put(consumer)


    def register_consumer(self, consumer: Consumer):
        if consumer in self.consumers:
            return
        self.consumers.append(consumer)
        consumer.task_done_with_product.connect(self.on_task_done_by_consumer)


class Consumer(QThread):
    task_done_with_product = Signal(object)
    def __init__(self, name: str, producer: Producer):
        self.name = name
        # Super init and set Thread name
        QThread.__init__(self, objectName=f'Thread_Of_{self.name}')
        self.producer = producer
        # See method on_product_available doc
        self.internal_queue = Queue()

    def run(self) -> None:
        self.producer.product_available.connect(self.on_product_available, Qt.ConnectionType.UniqueConnection)
        # Thread loop waiting for product availability
        product = self.internal_queue.get(block=True)
        print(f'{self.name}: Product {product} received and elaborated in thread {thread_name()}\n\n')

        # Tell the producer I've done
        self.task_done_with_product.emit(self)

        # Now the thread is naturally closed

    @Slot(list)
    def on_product_available(self, product: list):
        """
        As a limitation of PySide, it seems that list are not supported for QueuedConnection. This work around using
        internal queue might solve
        """
        # This is executed in Main Loop!
        print(f'{self.name}: In thread {thread_name()} I received the product, and I\'m queuing it for being elaborated'
              f'in consumer thread')
        self.internal_queue.put(product)
        # Quit the thread
        self.active = False

    def __repr__(self):
        # Needed in case of exception for representing current consumer
        return f'{self.name}'


# Needed to executed main and threads event loops
app = QCoreApplication()

QThread.currentThread().setObjectName('MainThread')

producer = Producer()

c1 = Consumer('Consumer1', producer)
c1.start()
producer.register_consumer(c1)

c2 = Consumer('Consumer2', producer)
c2.start()
producer.register_consumer(c2)

producer.product_available.connect(c1.on_product_available)
producer.product_available.connect(c2.on_product_available)

# Start Producer thread for LAST!
producer.start()

app.exec_()

結果：

Producer: from thread ThreadProducer I've sent my products
Consumer1: In thread MainThread I received the product, and I'm queuing it for being elaboratedin consumer thread
Consumer1: Product [0, 1, 2, 3, 4] received and elaborated in thread Thread_Of_Consumer1
Consumer2: In thread MainThread I received the product, and I'm queuing it for being elaboratedin consumer thread
Consumer2: Product [0, 1, 2, 3, 4] received and elaborated in thread Thread_Of_Consumer2
Producer: Consumer1 has told me it has completed his task with my product! (Thread ThreadProducer)
Producer: Consumer2 has told me it has completed his task with my product! (Thread ThreadProducer)
All consumers have completed their task! I'm terminating myself

筆記：

• 分步說明在代碼注釋中。 如果有什么不清楚的地方，我會盡力澄清
• 不幸的是，我還沒有找到一種使用QueueConnection （此處為文檔）的方法，以便直接將 Slot 執行到正確的線程中：內部隊列已用於將信息從主循環傳遞到正確的線程（生產者和消費者）。 似乎list和object不能在 PySide/pyqt 中進行元注冊以進行排隊