[英]Cython iterate over list of numpy arrays without the gil
I would like to iterate over a list of numpy arrays with different dimensions, and pass them in to a cython function that does not require the GIL: 我想遍历具有不同维数的numpy数组的列表,并将它们传递给不需要GIL的cython函数:
# a has T1 rows and M columns
a = np.array([[0.0, 0.1, 0.3, 0.7],
[0.1, 0.2, 0.1, 0.6],
[0.1, 0.2, 0.1, 0.6]])
# b has T2 rows and M columns
b = np.array([[1.0, 0.0, 0.0, 0.0],
[0.1, 0.2, 0.1, 0.6]])
# c has T3 rows and M columns
c = np.array([[0.1, 0.0, 0.3, 0.6],
[0.5, 0.2, 0.3, 0.0],
[0.0, 1.0, 0.0, 0.0],
[0.0, 0.0, 0.1, 0.0]])
array_list = [a, b, c]
N = len(array_list)
# this function works
@cython.boundscheck(False)
@cython.wraparound(False)
cdef void function_not_requiring_the_gil(double[:, ::1] arr) nogil:
cdef int T = arr.shape[0]
cdef int M = arr.shape[1]
cdef int i, t
for t in range(T):
for i in range(M):
# do some arbitrary thing to the array in-place
arr[t, i] += 0.001
# issue is with this function
def function_with_loop_over_arrays(array_list, int N):
cdef int i
with nogil:
for i in range(N):
function_not_requiring_the_gil(array_list[i])
When I compile the Cython code, I get the following error: 编译Cython代码时,出现以下错误:
Error compiling Cython file:
------------------------------------------------------------
...
def function_with_loop_over_arrays(array_list, int N):
cdef int i
with nogil:
for i in range(N):
function_not_requiring_the_gil(array_list[i]) ^
------------------------------------------------------------
my_file.pyx:312:53: Indexing Python object not allowed without gil
Is there another type of data structure that I can use instead of a Python list to store these numpy arrays so that I can iterate over them without the gil? 是否可以使用其他类型的数据结构代替Python列表来存储这些numpy数组,以便无需gil即可遍历它们? I'm open to suggestions involving malloc C pointers/Cython memoryviews/other types I'm not aware of. 我愿意接受涉及malloc C指针/ Cython memoryviews /其他我不知道的类型的建议。
Note that each numpy array has a different number of rows, but the length of the list of arrays is known. 请注意,每个numpy数组具有不同的行数,但是数组列表的长度是已知的。
You can pass two arrays of shape (3,)
to function_with_loop_over_arrays
: one ( array_starts
) which contains pointers to the first element of a
, b
and c
, and one ( arrays_rows
) which contains T1
, T2
and T3
. 您可以将两个形状为(3,)
数组传递给function_with_loop_over_arrays
:一个( array_starts
)包含指向a
, b
和c
的第一个元素的指针,另一个( arrays_rows
)包含T1
, T2
和T3
。
Then modify function_not_requiring_the_gil
so that it receives a pointer on the first element of the array and its number of rows, and you will be able to call it in function_with_loop_over_arrays
with: 然后修改function_not_requiring_the_gil
以便它在数组的第一个元素及其行数上收到一个指针,您将可以在function_with_loop_over_arrays
使用以下命令调用它:
for i in range(N): # N is 3 and should be passed to function_with_loop_over_arrays
function_not_requiring_the_gil(array_starts[i], array_rows[i])
As the error message says, you can't index a Python list
without the gil, and there aren't really any obvious alternative data-structures that fill the same role. 如错误消息所述,没有gil,您将无法为Python list
索引,并且实际上并没有任何明显的替代数据结构可以充当相同的角色。 You just need to move the nogil
to take the indexing outside it 您只需要移动nogil
即可将索引nogil
它
def function_with_loop_over_arrays(array_list, int N):
cdef int i
cdef double[:, ::1] tmp_mview
for i in range(N):
tmp_mview = array_list[i]
with nogil:
function_not_requiring_the_gil(tmp_mview)
(Equivalently you can put the indexing inside a with gil:
block instead.) (等效地,您可以with gil:
块将索引放入a内。)
There's a small cost to getting and releasing the gil
but provided your function_not_requiring_the_gil
does a decent amount of work compared to the indexing it should be insignificant. 获取和发布gil
成本很小,但前提是您的function_not_requiring_the_gil
所做的工作量可观,而对其进行的索引编制却微不足道。
nogil
features are often easy to control from numba : nogil
功能通常可以通过numba轻松控制:
import numba
@numba.jit(nogil=True)
def function_not_requiring_the_gil(arr):
T,N = arr.shape
for t in range(T):
for i in range(N):
# do some arbitrary thing to the array in-place
arr[t, i] += 0.001
def function_with_loop_over_arrays(array_list)
for a in array_list:
function_not_requiring_the_gil(a)
will give you the same (efficient) result. 将给您相同(有效)的结果。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.