[英]Python2: multiprocessing using a Pool of workers with multiple arguments
[英]Joining output of multiprocessing workers - python2
我在下面尝试了[抽象的]多处理脚本:
1)在两个进程之间分配工作负载(将1添加到列表中的每个变量+将新变量附加到新列表中)
2)将两个过程的输出合并到新的全局列表中,以供将来处理。
关于如何获取两个流程的输出并将该输出合并到全局列表中的任何想法? 在执行后,我最终想要得到的是:
new_id_list = [2、4、6、8、10、3、5、7、9、11]#worker1的#new_id_list + worker2的new_id_list
#python2
from multiprocessing import Process, Queue
from time import sleep
#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]
#new output id list
new_id_list = []
queue = Queue()
def mp_worker(queue):
while queue.qsize() >0 :
record = queue.get()
new_id_list.append(record+1)
sleep(.1)
print(new_id_list)
###how would I go about passing this new_id_list as the global variable
print("worker closed")
def mp_handler():
# Spawn two processes, assigning the method to be executed
# and the input arguments (the queue)
processes = [Process(target=mp_worker, args=(queue,)) for _ in range(2)]
for process in processes:
process.start()
print('Process started')
for process in processes:
process.join()
if __name__ == '__main__':
for id in id_list:
queue.put(id)
mp_handler()
我认为您遇到的问题是两个进程无法共享new_id_list
。
你需要做的是创建另一个Queue
,这将代表结果队列,并把它传递给这两个进程。 根据需要在进程内部添加到队列,在两个进程执行结束时(在process.join()
),您只需从列表中的队列中提取所有内容即可。
下面的工作代码。 基本上:
1)我们使用multiprocessing.Manager()
2)使用管理器生成列表
3)将列表传递给每个工作人员,然后让每个工作人员将输出附加回列表中。
from multiprocessing import Process, Queue
from time import sleep
import multiprocessing
#records to process
id_list = [1,2,3,4,5,6,7,8,9,10]
#new output id list
new_id_list = []
queue = Queue()
def mp_worker(d,queue):
while queue.qsize() >0 :
record = queue.get()
new_id_list.append(record+1)
d.append(record+1)
sleep(.1)
print(new_id_list)
print("worker closed")
def mp_handler():
# Spawn two processes, assigning the method to be executed
# and the input arguments (the queue)
processes = [Process(target=mp_worker, args=(d,queue,)) for _ in range(2)]
for process in processes:
process.start()
print('Process started')
for process in processes:
process.join()
if __name__ == '__main__':
mgr = multiprocessing.Manager()
d = mgr.list()
for id in id_list:
queue.put(id)
mp_handler()
print(d) #
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.