[英]Python 2.6: Process local storage while using multiprocessing.Pool
我正在嘗試構建一個Python腳本,該腳本具有跨大量數據集的工作進程池(使用mutiprocessing.Pool)。
我希望每個進程都有一個唯一的對象,該對象將在該進程的多次執行中使用。
偽代碼:
def work(data):
#connection should be unique per process
connection.put(data)
print 'work done with connection:', connection
if __name__ == '__main__':
pPool = Pool() # pool of 4 processes
datas = [1..1000]
for process in pPool:
#this is the part i'm asking about // how do I really do this?
process.connection = Connection(conargs)
for data in datas:
pPool.apply_async(work, (data))
我認為類似的方法應該有效(未經測試)
def init(*args):
global connection
connection = Connection(*args)
pPool = Pool(initializer=init, initargs=conargs)
直接創建mp.Process
es(沒有mp.Pool
)可能是最簡單的:
import multiprocessing as mp
import time
class Connection(object):
def __init__(self,name):
self.name=name
def __str__(self):
return self.name
def work(inqueue,conn):
name=mp.current_process().name
while 1:
data=inqueue.get()
time.sleep(.5)
print('{n}: work done with connection {c} on data {d}'.format(
n=name,c=conn,d=data))
inqueue.task_done()
if __name__ == '__main__':
N=4
procs=[]
inqueue=mp.JoinableQueue()
for i in range(N):
conn=Connection(name='Conn-'+str(i))
proc=mp.Process(target=work,name='Proc-'+str(i),args=(inqueue,conn))
proc.daemon=True
proc.start()
datas = range(1,11)
for data in datas:
inqueue.put(data)
inqueue.join()
產量
Proc-0: work done with connection Conn-0 on data 1
Proc-1: work done with connection Conn-1 on data 2
Proc-3: work done with connection Conn-3 on data 3
Proc-2: work done with connection Conn-2 on data 4
Proc-0: work done with connection Conn-0 on data 5
Proc-1: work done with connection Conn-1 on data 6
Proc-3: work done with connection Conn-3 on data 7
Proc-2: work done with connection Conn-2 on data 8
Proc-0: work done with connection Conn-0 on data 9
Proc-1: work done with connection Conn-1 on data 10
請注意, Proc
編號每次都對應於相同的Conn
編號。
進程本地存儲很容易實現為映射容器,對於任何其他從Google來這里的人來說,尋找類似的東西(請注意,這是Py3,但很容易轉換為2的語法(僅繼承自object
):
class ProcessLocal:
"""
Provides a basic per-process mapping container that wipes itself if the current PID changed since the last get/set.
Aka `threading.local()`, but for processes instead of threads.
"""
__pid__ = -1
def __init__(self, mapping_factory=dict):
self.__mapping_factory = mapping_factory
def __handle_pid(self):
new_pid = os.getpid()
if self.__pid__ != new_pid:
self.__pid__, self.__store = new_pid, self.__mapping_factory()
def __delitem__(self, key):
self.__handle_pid()
return self.__store.__delitem__(key)
def __getitem__(self, key):
self.__handle_pid()
return self.__store.__getitem__(key)
def __setitem__(self, key, val):
self.__handle_pid()
return self.__store.__setitem__(key)
查看更多@ https://github.com/akatrevorjay/pytutils/blob/develop/pytutils/mappings.py
您想讓一個對象駐留在共享內存中,對嗎?
Python在其標准庫中對此提供了一些支持,但是有點差。 據我所知,只能存儲整數和其他一些原始類型。
嘗試POSH(Python對象共享): http : //poshmodule.sourceforge.net/
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.