[英]Multiprocessing using shared memory
Can someone provide me with sample code to share a writable array or a list among a pool of worker processes or even individually spawned processes using the multiprocessing module of python using locks? 有人可以给我提供示例代码,以便在使用锁的python多处理模块中,在一组工作进程甚至单个生成的进程之间共享可写数组或列表吗? My code below spawns 2 processes of which one should print '1' and the other should print '2' to the shared array.
我下面的代码产生了2个进程,其中一个应向共享数组打印“ 1”,另一个应向共享数组打印“ 2”。 However, when I try to print out the elements of the array after the processing it only gives me a list of 0's.
但是,当我尝试在处理后打印出数组的元素时,它只会显示0列表。 Where am I going wrong?
我要去哪里错了? I want a writable data structure to be shared between multiple processes.
我希望在多个进程之间共享可写的数据结构。
Below is my code: 下面是我的代码:
import multiprocessing
arr=multiprocessing.Array('i',10,lock=True)
lock=multiprocessing.RLock()
def putitin(n):
for i in range(5):
lock.acquire()
arr.append(n)
lock.release()
return
p1=multiprocessing.Process(target=putitin,args=(1,))
p2=multiprocessing.Process(target=putitin,args=(2,))
p1.start()
p2.start()
p1.join()
p2.join()
for i in range(10):
print(arr[i])
One potential problem with your code is that to use multiprocessing
on Windows you need to put the code for the main process in an if __name__ == '__main__':
block. 代码的一个潜在问题是,要在Windows上使用
multiprocessing
,您需要将主进程的代码放在if __name__ == '__main__':
块中。 See the Safe importing of main module subsection of the Windows section of the multiprocessing Programming guidelines . 请参阅《 多处理编程指南 》的Windows部分的“ 安全导入主模块”小节。
Another major one is that you're attempting to a share global variable among the processes. 另一个主要问题是您正在尝试在流程之间共享全局变量。 which won't work because each one runs in its own unshared memory-space, so each subprocess has it's own
arr
. 这将不起作用,因为每个进程都在自己的未共享内存空间中运行,因此每个子进程都有自己的
arr
。 (Variables which are just module level constants are OK, though) (不过,只是模块级常量的变量是可以的)
Lastly, a multiprocessing.Array
has a fixed size and doesn't have the extend()
method your code is trying to use in the putitin()
function — therefore it appears you also want a writable and resizable container (that is ordered and perhaps accessible via integer indices). 最后,
multiprocessing.Array
具有固定的大小,并没有extend()
你的代码试图在使用这种方法putitin()
函数-因此看来你也希望有一个可以写入的可调整大小的容器(即下令,也许可通过整数索引访问)。
In that case something the like the following may be suitable. 在这种情况下,类似以下的内容可能是合适的。 You don't need to explicitly lock the object before make changes to it because it's a thread-safe shared object.
您无需在更改对象之前显式锁定对象,因为它是线程安全的共享对象。
import multiprocessing
def putitin(lst, value):
for i in range(5):
lst.append(value)
if __name__ == '__main__':
manager = multiprocessing.Manager()
lst = manager.list()
p1 = multiprocessing.Process(target=putitin, args=(lst, 1))
p2 = multiprocessing.Process(target=putitin, args=(lst, 2))
p1.start()
p2.start()
p1.join()
p2.join()
for i in range(len(lst)):
print(lst[i])
Output: 输出:
1
1
1
1
1
2
2
2
2
2
A few issues that I found in your code. 我在您的代码中发现了一些问题。 First of all, it seems to be a good idea to pass all shared resources to child and use
if __name__ == '__main__'
. 首先,将所有共享资源传递给child并使用
if __name__ == '__main__'
似乎是一个好主意 。 Secondly, I don't think multiprocessing.Array
has append()
method (at least it didn't work for me on Python2 and Python3). 其次,我不认为
multiprocessing.Array
具有append()
方法(至少在Python2和Python3上对我不起作用)。 Thirdly, since you are using lock=True
I don't think you need additional locks. 第三,由于您使用的是
lock=True
所以我认为您不需要其他锁。
If you need to append values to the Array I'd use separate Counter variable (see the code below). 如果需要将值附加到数组,则可以使用单独的Counter变量(请参见下面的代码)。
import multiprocessing
def putitin(n, arr, counter):
for i in range(5):
with counter.get_lock():
index = counter.value
counter.value += 1
arr[index] = n
if __name__ == '__main__':
arr = multiprocessing.Array('i', 10,lock=True)
counter = multiprocessing.Value('i', 0, lock=True)
p1 = multiprocessing.Process(target=putitin,args=(1, arr, counter))
p2 = multiprocessing.Process(target=putitin,args=(2, arr, counter))
p1.start()
p2.start()
p2.join()
p1.join()
for i in range(10):
print(arr[i])
Having said that, what is your high-level goal? 话虽如此,您的高层目标是什么? What are you trying to achieve with it?
您正在尝试实现什么? Maybe it's possible to use
multiprocessing.Pool
? 也许可以使用
multiprocessing.Pool
?
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.