[英]Python passing queue objects to multiprocessing.Pool of processes
I have a shared multiprocessing.Pool
object in my application which is initialized with two queue objects (one for jobs and the other for results). 我的应用程序中有一个共享的multiprocessing.Pool
对象,该对象用两个队列对象(一个用于作业,另一个用于结果)初始化。
How can I send an arbitrary queue object into the job queue and have the process send the result to this alternative queue? 如何将任意队列对象发送到作业队列中,并使进程将结果发送到此备用队列?
job_q.put_nowait((item, alt_q)) # Raises an exception.
This approach works fine when doing multithreading but not when doing multiprocessing. 这种方法在执行多线程处理时效果很好,但在执行多处理时效果不佳。
The example code below demonstrates what I am trying to achieve. 下面的示例代码演示了我正在尝试实现的目标。 I initialize the pool with two multiprocessing.Queue objects job_q
and res_q
. 我使用两个multiprocessing.Queue对象job_q
和res_q
初始化池。 Well, in fact, they are proxies created by multiprocessing.Manager. 好吧,实际上,它们是由multiprocessing.Manager创建的代理。 The run
function is the run-loop of each process, it monitors the job queue for items and simply adds the items to the result queue. run
功能是每个进程的运行循环,它监视作业队列中的项目并将其简单地添加到结果队列中。 (A separate thread is monitoring the result queue and print to stdout). (一个单独的线程正在监视结果队列并打印到stdout)。
import multiprocessing as mp
import queue
import threading
import time
import os
def run(job_queue, result_queue):
""" Run-loop for each process.
"""
print("Starting process {}".format(os.getpid()))
while True:
job_q = job_queue
res_q = result_queue
try:
# `item` is just a string
# `opt_queue` is an optional result queue to use
item, opt_queue = job_q.get(True, 0.05)
if opt_queue is not None:
res_q = opt_queue
item = item + " Processed"
res_q.put_nowait(item)
except queue.Empty:
continue
def monitor_queue(mp_queue):
""" The target of a monitoring thread.
"""
while True:
try:
item = mp_queue.get(True, 0.05)
print("Got `{}`".format(item))
except queue.Empty:
continue
if __name__ == '__main__':
m = mp.Manager()
job_q = m.Queue()
res_q = m.Queue()
alt_q = m.Queue()
# Monitor `res_q` for items
threading.Thread(target=monitor_queue, args=(res_q,)).start()
# Monitor `alt_q` for items
threading.Thread(target=monitor_queue, args=(alt_q,)).start()
# `run` is called by each process, share `job_q` and `res_q` with all processes
pool = mp.Pool(2, run, (job_q, res_q))
time.sleep(1)
# Add an item to `job_q` and `None` means send result to `res_q`
print('Putting first item into the job queue')
job_q.put_nowait(('#1', None)) # prints... Got `#1`
time.sleep(1)
# Add an item to `job_q` and send result to `alt_q`
print('Putting second item into the job queue and passing alternative result queue')
job_q.put_nowait(('#2', alt_q)) # TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
pool.close()
pool.terminate()
This exits with the error 这退出并显示错误
Putting second item into the job queue and passing alternative result queue
Traceback (most recent call last):
File "/Users/daniel/Desktop/pydebug/mp_example.py", line 54, in <module>
job_q.put_nowait(('#1', alt_q)) # TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
File "<string>", line 2, in put_nowait
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/managers.py", line 772, in _callmethod
raise convert_to_error(kind, result)
multiprocessing.managers.RemoteError:
---------------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/managers.py", line 228, in serve_client
request = recv()
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/connection.py", line 251, in recv
return _ForkingPickler.loads(buf.getbuffer())
File "/usr/local/Cellar/python3/3.6.1/Frameworks/Python.framework/Versions/3.6/lib/python3.6/multiprocessing/managers.py", line 881, in RebuildProxy
return func(token, serializer, incref=incref, **kwds)
TypeError: AutoProxy() got an unexpected keyword argument 'manager_owned'
---------------------------------------------------------------------------
我认为您无法在消息中传递队列,因为它不可序列化。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.