多處理Python中的共享數組

Question

我試圖在並行處理python腳本中的同一共享數組中編寫。

當我在課外使用普通腳本進行操作時，一切正常。 但是，當我嘗試通過一個類（使用相同的代碼）進行操作時，我得到了
Runtime Error: SynchronizedArray objects should only be shared between processes through inheritance 。

我的腳本如下（沒有類）：

import numpy
import ctypes

from multiprocessing import Pool, Array, cpu_count

n = 2

total_costs_matrix_base = Array(ctypes.c_double, n*n)
total_costs_matrix = numpy.ctypeslib.as_array(
                     total_costs_matrix_base.get_obj())
total_costs_matrix = total_costs_matrix.reshape(n,n)


def set_total_costs_matrix( i, j, def_param = total_costs_matrix_base):
    total_costs_matrix[i,j] = i * j

if __name__ == "__main__":

    pool = Pool(processes=cpu_count())
    iterable = []

    for i in range(n):
        for j in range(i+1,n):
            iterable.append((i,j))
    pool.starmap(set_total_costs_matrix, iterable)
    total_costs_matrix.dump('some/path/to/file')

該腳本運行良好。 以下是不使用的（使用類）：

import numpy
import ctypes

from multiprocessing import Pool, Array, cpu_count

class CostComputation(object):
    """Computes the cost matrix."""

    def __init__(self):
        self.n = 2

        self.total_costs_matrix_base = Array(ctypes.c_double, self.n*self.n)
        self.total_costs_matrix = numpy.ctypeslib.as_array(
                             self.total_costs_matrix_base.get_obj())
        self.total_costs_matrix = self.total_costs_matrix.reshape(self.n,self.n)


    def set_total_costs_matrix(self, i, j, def_param = None):
        def_param = self.total_costs_matrix_base
        self.total_costs_matrix[i,j] = i * j


    def write_cost_matrix(self):
        pool = Pool(processes=cpu_count())
        iterable = []

        for i in range(self.n):
            for j in range(i+1,self.n):
                iterable.append((i,j))
        pool.starmap(self.set_total_costs_matrix, iterable)
        self.total_costs_matrix.dump('some/path/to/file')

在此之后，我會打電話write_cost_matrix從另一個文件創建的實例后CostComputation 。

我讀了這個答案，但仍然無法解決我的問題。

我在Mac OSX Yosemite 10.10.4中使用Python 3.4.2。

編輯
使用類CostComputation時，我正在使用的腳本是：

from cost_computation import CostComputation

cc = CostComputation()
cc.write_costs_matrix()

整個錯誤是：

Traceback (most recent call last):
  File "app.py", line 65, in <module>
    cc.write_cost_matrix()
  File "/path/to/cost_computation.py", line 75, in write_cost_matrix
    pool.starmap(self.set_total_costs_matrix, iterable)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 268, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 599, in get
    raise self._value
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/pool.py", line 383, in _handle_tasks
    put(task)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/connection.py", line 206, in send
    self._send_bytes(ForkingPickler.dumps(obj))
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/reduction.py", line 50, in dumps
    cls(buf, protocol).dump(obj)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/sharedctypes.py", line 192, in __reduce__
    assert_spawning(self)
  File "/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/multiprocessing/context.py", line 347, in assert_spawning
    ' through inheritance' % type(obj).__name__
RuntimeError: SynchronizedArray objects should only be shared between processes through inheritance

Answer 1

嘗試創建僅包含共享數據的第二個類。 然后在您的主類中使用該類對象。