[英]Pool with permanent objects for each worker without “global” keyword
I am using a Pool to benefit of multiple cores. 我使用池来获益多核。 Each worker in the pool needs its own Calculator object. 池中的每个worker都需要自己的Calculator对象。 The initialization of calculator is quite time consuming, so I would like to have it generated only once per worker in the pool and not every time, a new task arrives. 计算器的初始化非常耗时,所以我想让它在池中的每个工人只生成一次,而不是每次都有新的任务到来。 The only way, I got this working was by using the “ugly“ keyword global
. 唯一的方法,我得到的工作是使用“丑陋”关键字global
。 Is there a “cleaner” way to implement this? 是否有一种“更清洁”的方式来实现这一点?
I would like to avoid queues (parent thread is often sigkill'd and leaves child processes when using queues) and managers (performance too slow). 我想避免队列(父线程通常是sigkill'd并在使用队列时留下子进程)和管理器(性能太慢)。
#!/usr/bin/python
# -*- coding: utf-8 -*-
import multiprocessing
def init_pool():
global calculator
calculator = Calculator() # should only executed ones per worker
def run_pool(args):
return calculator.calculate(*args) # time consuming calculation
class Organiser():
def __init__(self):
self.__pool = multiprocessing.Pool(initializer=init_pool)
def process(self, tasks):
results = self.__pool.map(run_pool, tasks)
return results
I don't see a way to achieve what you want (initialize exactly once per worker). 我没有看到实现你想要的方法(每个工人只准备一次)。
But the following seems to work if you want to initialize "Calculator" exactly once for the whole group of workers. 但是,如果您想为整个工作组初始化一次“计算器”,则以下似乎可行。
def run_pool(args):
calculator,arg = args
return calculator.calculate(arg) # time consuming calculation
class Organiser():
def __init__(self):
self.calculator = Calculator()
self.__pool = multiprocessing.Pool(processes=4)
def process(self, tasks):
results = self.__pool.map(run_pool, [(self.calculator,data) for data in tasks])
return results
To initialize exactly once per worker, it appears to me that you must use global variables or singletons (equivalent). 要为每个工人初始化一次,在我看来,您必须使用全局变量或单例(等效)。 I will await other answers to your question as well :) 我将等待你的问题的其他答案:)
Regards, Siddharth 此致,Siddharth
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.