![](/img/trans.png)
[英]IndexError: tuple index out of range, when trying to pass 2d array in multiprocessing.Pool
[英]How to pass 2d array as multiprocessing.Array to multiprocessing.Pool?
我的目標是將父數組傳遞給mp.Pool
並用2
s 填充它,同時將其分發到不同的進程。 這適用於一維數組:
import numpy as np
import multiprocessing as mp
import itertools
def worker_function(i=None):
global arr
val = 2
arr[i] = val
print(arr[:])
def init_arr(arr=None):
globals()['arr'] = arr
def main():
arr = mp.Array('i', np.zeros(5, dtype=int), lock=False)
mp.Pool(1, initializer=init_arr, initargs=(arr,)).starmap(worker_function, zip(range(5)))
print(arr[:])
if __name__ == '__main__':
main()
輸出:
[2, 0, 0, 0, 0]
[2, 2, 0, 0, 0]
[2, 2, 2, 0, 0]
[2, 2, 2, 2, 0]
[2, 2, 2, 2, 2]
[2, 2, 2, 2, 2]
但是我怎樣才能對 x 維數組做同樣的事情呢? 向arr
添加維度:
arr = mp.Array('i', np.zeros((5, 5), dtype=int), lock=False)
產生一個錯誤:
Traceback (most recent call last):
File "C:/Users/Artur/Desktop/RL_framework/test2.py", line 23, in <module>
main()
File "C:/Users/Artur/Desktop/RL_framework/test2.py", line 17, in main
arr = mp.Array('i', np.zeros((5, 5), dtype=int), lock=False)
File "C:\Users\Artur\anaconda3\envs\RL_framework\lib\multiprocessing\context.py", line 141, in Array
ctx=self.get_context())
File "C:\Users\Artur\anaconda3\envs\RL_framework\lib\multiprocessing\sharedctypes.py", line 88, in Array
obj = RawArray(typecode_or_type, size_or_initializer)
File "C:\Users\Artur\anaconda3\envs\RL_framework\lib\multiprocessing\sharedctypes.py", line 67, in RawArray
result.__init__(*size_or_initializer)
TypeError: only size-1 arrays can be converted to Python scalars
更改dtype
的arr
也沒有幫助。
您不能直接將multiprocessing.Array
用作二維數組,但在一維內存中,無論如何,第二維只是一種錯覺:)。
幸運的是 numpy 允許從緩沖區讀取數組並對其進行整形,而無需復制它。 在下面的演示中,我只使用了一個單獨的鎖,這樣我們就可以逐步觀察所做的更改,目前沒有競爭條件。
import multiprocessing as mp
import numpy as np
def worker_function(i):
global arr, arr_lock
val = 2
with arr_lock:
arr[i, :i+1] = val
print(f"{mp.current_process().name}\n{arr[:]}")
def init_arr(arr, arr_lock=None):
globals()['arr'] = np.frombuffer(arr, dtype='int32').reshape(5, 5)
globals()['arr_lock'] = arr_lock
def main():
arr = mp.Array('i', np.zeros(5 * 5, dtype='int32'), lock=False)
arr_lock = mp.Lock()
mp.Pool(2, initializer=init_arr, initargs=(arr, arr_lock)).map(
worker_function, range(5)
)
arr = np.frombuffer(arr, dtype='int32').reshape(5, 5)
print(f"{mp.current_process().name}\n{arr}")
if __name__ == '__main__':
main()
輸出:
ForkPoolWorker-1
[[2 0 0 0 0]
[0 0 0 0 0]
[0 0 0 0 0]
[0 0 0 0 0]
[0 0 0 0 0]]
ForkPoolWorker-2
[[2 0 0 0 0]
[2 2 0 0 0]
[0 0 0 0 0]
[0 0 0 0 0]
[0 0 0 0 0]]
ForkPoolWorker-1
[[2 0 0 0 0]
[2 2 0 0 0]
[2 2 2 0 0]
[0 0 0 0 0]
[0 0 0 0 0]]
ForkPoolWorker-2
[[2 0 0 0 0]
[2 2 0 0 0]
[2 2 2 0 0]
[2 2 2 2 0]
[0 0 0 0 0]]
ForkPoolWorker-1
[[2 0 0 0 0]
[2 2 0 0 0]
[2 2 2 0 0]
[2 2 2 2 0]
[2 2 2 2 2]]
MainProcess
[[2 0 0 0 0]
[2 2 0 0 0]
[2 2 2 0 0]
[2 2 2 2 0]
[2 2 2 2 2]]
Process finished with exit code 0
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.