[英]Passing list of numpy arrays to C using cython
I have a list list_of_arrays
of 3D numpy arrays that I want to pass to a C function with the template我有一个 3D numpy 数组的列表
list_of_arrays
,我想用模板传递给 C 函数
int my_func_c(double **data, int **shape, int n_arrays)
such that以至于
data[i] : pointer to the numpy array values in list_of_arrays[i]
shape[i] : pointer to the shape of the array in list_of_arrays[i] e.g. [2,3,4]
How can I call my_func_c
using a cython interface function?如何使用 cython 接口函数调用
my_func_c
?
My first idea was to do something like below (which works) but I feel there is a better way just using numpy arrays without mallocing and freeing.我的第一个想法是做类似下面的事情(可行),但我觉得有一种更好的方法,只使用 numpy 数组而不进行 malloc 和释放。
# my_func_c.pyx
import numpy as np
cimport numpy as np
cimport cython
from libc.stdlib cimport malloc, free
cdef extern from "my_func.c":
double my_func_c(double **data, int **shape, int n_arrays)
def my_func(list list_of_arrays):
cdef int n_arrays = len(list_of_arrays)
cdef double **data = <double **> malloc(n_arrays*sizeof(double *))
cdef int **shape = <int **> malloc(n_arrays*sizeof(int *))
cdef double x;
cdef np.ndarray[double, ndim=3, mode="c"] temp
for i in range(n_arrays):
temp = list_of_arrays[i]
data[i] = &temp[0,0,0]
shape[i] = <int *> malloc(3*sizeof(int))
for j in range(3):
shape[i][j] = list_of_arrays[i].shape[j]
x = my_func_c(data, shape, n_arrays)
# Free memory
for i in range(n_arrays):
free(shape[i])
free(data)
free(shape)
return x
NB NB
To see a working example we can use a very simple function calculating the product of all the arrays in our list.要查看一个工作示例,我们可以使用一个非常简单的函数来计算列表中所有数组的乘积。
# my_func.c
double my_func_c(double **data, int **shape, int n_arrays) {
int array_idx, i0, i1, i2;
double prod = 1.0;
// Loop over all arrays
for (array_idx=0; array_idx<n_arrays; array_idx++) {
for (i0=0; i0<shape[array_idx][0]; i0++) {
for (i1=0; i1<shape[array_idx][1]; i1++) {
for (i2=0; i2<shape[array_idx][2]; i2++) {
prod = prod*data[array_idx][i0*shape[array_idx][1]*shape[array_idx][2] + i1*shape[array_idx][2] + i2];
}
}
}
}
return prod;
}
Create the setup.py
file,创建
setup.py
文件,
# setup.py
from distutils.core import setup
from Cython.Build import cythonize
import numpy as np
setup(
name='my_func',
ext_modules = cythonize("my_func_c.pyx"),
include_dirs=[np.get_include()]
)
Compile编译
python3 setup.py build_ext --inplace
Finally we can run a simple test最后我们可以运行一个简单的测试
# test.py
import numpy as np
from my_func_c import my_func
a = [1+np.random.rand(3,1,2), 1+np.random.rand(4,5,2), 1+np.random.rand(1,2,3)]
print('Numpy product: {}'.format(np.prod([i.prod() for i in a])))
print('my_func product: {}'.format(my_func(a)))
using使用
python3 test.py
One alternative would be to let numpy manage your memory for you.一种替代方法是让 numpy 为您管理您的记忆。 You can do this by using numpy arrays of
np.uintp
which is an unsigned int with the same size as any pointer.您可以通过使用
np.uintp
numpy 数组来做到这一点,它是一个与任何指针大小相同的无符号整数。
Unfortunately, this does require some type-casting (between "pointer sized int" and pointers) which is a good way of hiding logic errors, so I'm not 100% happy with it.不幸的是,这确实需要一些类型转换(在“指针大小的 int”和指针之间),这是隐藏逻辑错误的好方法,所以我对它不是 100% 满意。
def my_func(list list_of_arrays):
cdef int n_arrays = len(list_of_arrays)
cdef np.uintp_t[::1] data = np.array((n_arrays,),dtype=np.uintp)
cdef np.uintp_t[::1] shape = np.array((n_arrays,),dtype=np.uintp)
cdef double x;
cdef np.ndarray[double, ndim=3, mode="c"] temp
for i in range(n_arrays):
temp = list_of_arrays[i]
data[i] = <np.uintp_t>&temp[0,0,0]
shape[i] = <np.uintp_t>&(temp.shape[0])
x = my_func_c(<double**>(&data[0]), <np.intp_t**>&shape[0], n_arrays)
(I should point out that I've only confirmed that it compiles and not tested it further, but the basic idea should be OK) (我应该指出,我只确认它编译并没有进一步测试它,但基本思想应该没问题)
The way you've done it is probably a pretty sensible way.你这样做的方式可能是一种非常明智的方式。 One slight simplification to your original code that should work
对应该可以工作的原始代码稍作简化
shape[i] = <np.uintp_t>&(temp.shape[0])
instead of malloc
and copy.而不是
malloc
和 copy。 I'd also recommend putting the free
s in a finally
block to ensure they get run.我还建议将
free
放在finally
块中以确保它们运行。
Edit: @ead has helpfully pointed out that the numpy shape is stored as as np.intp_t
- ie an signed integer big enough to fit a pointer in, which is mostly 64bit - while int
is usually 32 bit.编辑: @ead 有帮助地指出numpy 形状存储为
np.intp_t
- 即一个足以容纳指针的有符号整数,主要是 64 位 - 而int
通常是 32 位。 Therefore, to pass the shape without copying you'd need to change your C api.因此,要在不复制的情况下传递形状,您需要更改 C api。 Casting help makes that mistake harder to spot ("a good way of hiding logic errors")
转换帮助使该错误更难以发现(“隐藏逻辑错误的好方法”)
I think this is a good pattern to consume C-functionality from C++-code, and it can be also used here and would have two advantages:我认为这是从 C++ 代码中使用 C 功能的一个很好的模式,它也可以在这里使用,并且有两个优点:
To solve your problems you could use std::vector
:要解决您的问题,您可以使用
std::vector
:
import numpy as np
cimport numpy as np
from libcpp.vector cimport vector
cdef extern from "my_func.c":
double my_func_c(double **data, int **shape, int n_arrays)
def my_func(list list_of_arrays):
cdef int n_arrays = len(list_of_arrays)
cdef vector[double *] data
cdef vector [vector[int]] shape_mem # for storing casted shapes
cdef vector[int *] shape #pointers to stored shapes
cdef double x
cdef np.ndarray[double, ndim=3, mode="c"] temp
shape_mem.resize(n_arrays)
for i in range(n_arrays):
print "i:", i
temp = list_of_arrays[i]
data.push_back(&temp[0,0,0])
for j in range(3):
shape_mem[i].push_back(temp.shape[j])
shape.push_back(shape_mem[i].data())
x = my_func_c(data.data(), shape.data(), n_arrays)
return x
Also your setup would need a modification:您的设置也需要修改:
# setup.py
from distutils.core import setup, Extension
from Cython.Build import cythonize
import numpy as np
setup(ext_modules=cythonize(Extension(
name='my_func_c',
language='c++',
extra_compile_args=['-std=c++11'],
sources = ["my_func_c.pyx", "my_func.c"],
include_dirs=[np.get_include()]
)))
I prefer to use std::vector.data()
over &data[0]
because the second would mean undefined behavior for empty data
, and that is the reason we need std=c++11
flag.我更喜欢使用
std::vector.data()
不是&data[0]
因为第二个意味着空data
未定义行为,这就是我们需要std=c++11
标志的原因。
But in the end, it is for you to decide, which trade-off to make: the additional complexity of C++ (it has it own pitfalls) vs. handmade memory management vs. letting go of type safety for a short moment.但最终,由您来决定,进行哪种权衡:C++ 的额外复杂性(它有自己的陷阱)与手工内存管理与暂时放弃类型安全。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.