在Python多处理中的池进程之间传递消息

Question

I am new to python multiprocessing. I am writing a redis pubsub module in which a publisher reads data from a database table and publishes the data on a channel. At subscribers side I intend to create a number of worker processes which can listen to the channel by subscribing to it, get the item from channel and perform some operation on it. I want to stop all pool processes when any process gets a END string from channel. For this I am using multiprocessing.Manager().Queue() to communicate among pool processes. My subscriber module is below:

import redis
import time
import traceback
import json
import multiprocessing as mp
import contextlib

NO_OF_PROCESS = 4

def newProcess(queue):
    r = redis.StrictRedis()
    p = r.pubsub()
    p.subscribe('subChannel')

    while True:
        print ('Waiting...')
        if (queue.get() == 'END'):
            break 
        # Do some work from message
        message = p.get_message()
        print (message)
        if message:
            data = message['data']
            # If 'END' received than stop all pool processes by setting a message in shared queue
            if data == b'END':
                queue.put('END')
                break
            else:
                data = json.loads(data)
                print (data)
       time.sleep(1)


class Worker:
    def __init__(self, NO_OF_PROCESS):
        self.NO_OF_PROCESS = NO_OF_PROCESS
        self.queue = mp.Manager().Queue()
        self.pool = mp.Pool(processes = NO_OF_PROCESS)

    def doWork(self):
        [self.pool.apply(newProcess, args=(self.queue)) for i in range(self.NO_OF_PROCESS)]



def parallelExec():
    worker = Worker(NO_OF_PROCESS)
    worker.doWork()

parallelExec()

The problem here is I am getting following error when I try to execute: TypeError: newProcess() argument after * must be an iterable, not AutoProxy[Queue] . Please help, also if there is any other approach than I would like to hear.

Answer 1

您不是以可迭代的方式发送args，通过添加逗号将args转换为元组应该可以解决此问题

        [self.pool.apply(newProcess, args=(self.queue,)) for i in range(self.NO_OF_PROCESS)]