[英]Sharing large arbitrary data structures during multiprocessing
I would like to parallelize a process in python which needs read access to several large, non-array data structures.我想在 python 中并行化一个需要对几个大型非数组数据结构进行读取访问的进程。 What would be a recommended way to do this without copying all of the large data structures into every new process?在不将所有大型数据结构复制到每个新进程中的情况下,推荐的方法是什么?
Thank you谢谢
The multiprocessing package provides two ways of sharing state: shared memory objects and server process managers . multiprocessing 包提供了两种共享状态的方式:共享内存对象和服务器进程管理器。 You should use server process managers as they support arbitrary object types.您应该使用服务器进程管理器,因为它们支持任意对象类型。
The following program makes use of a server process manager:以下程序使用服务器进程管理器:
#!/usr/bin/env python3
from multiprocessing import Process, Manager
# Simple data structure
class DataStruct:
data_id = None
data_str = None
def __init__(self, data_id, data_str):
self.data_id = data_id
self.data_str = data_str
def __str__(self):
return f"{self.data_str} has ID {self.data_id}"
def __repr__(self):
return f"({self.data_id}, {self.data_str})"
def set_data_id(self, data_id):
self.data_id = data_id
def set_data_str(self, data_str):
self.data_str = data_str
def get_data_id(self):
return self.data_id
def get_data_str(self):
return self.data_str
# Create function to manipulate data
def manipulate_data_structs(data_structs, find_str):
for ds in data_structs:
if ds.get_data_str() == find_str:
print(ds)
# Create manager context, modify the data
with Manager() as manager:
# List of DataStruct objects
l = manager.list([
DataStruct(32, "Andrea"),
DataStruct(45, "Bill"),
DataStruct(21, "Claire"),
])
# Processes that look for DataStructs with a given String
procs = [
Process(target = manipulate_data_structs, args = (l, "Andrea")),
Process(target = manipulate_data_structs, args = (l, "Claire")),
Process(target = manipulate_data_structs, args = (l, "David")),
]
for proc in procs:
proc.start()
for proc in procs:
proc.join()
For more information, see Sharing state between processes in the documentation.有关更多信息,请参阅文档中的进程间共享状态。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.