[英]Passing C++ vector to Numpy through Cython without copying and taking care of memory management automatically
Dealing with processing large matrices (NxM with 1K <= N <= 20K & 10K <= M <= 200K), I often need to pass Numpy matrices to C++ through Cython to get the job done and this works as expected & without copying.处理大型矩阵(NxM with 1K <= N <= 20K & 10K <= M <= 200K),我经常需要通过 Cython 将 Numpy 矩阵传递给 C++ 以完成工作,这按预期工作且无需复制。
However , there are times when I need to initiate and preprocess a matrix in C++ and pass it to Numpy (Python 3.6) .但是,有时我需要在 C++ 中启动和预处理矩阵并将其传递给Numpy (Python 3.6) 。 Let's assume the matrices are linearized (so the size is N*M and it's a 1D matrix - col/row major doesn't matter here).让我们假设矩阵是线性化的(所以大小是 N*M 并且它是一维矩阵 - col/row major 在这里无关紧要)。 Following the information in here: exposing C-computed arrays in Python without data copies & modifying it for C++ compatibility, I'm able to pass C++ array.按照此处的信息:在没有数据副本的情况下在 Python 中公开 C 计算数组并修改它以实现 C++ 兼容性,我能够传递 C++ 数组。
The problem is if I want to use std vector instead of initiating array, I'd get Segmentation fault.问题是如果我想使用 std vector 而不是初始化数组,我会得到 Segmentation fault。 For example, considering the following files:例如,考虑以下文件:
fast.h快.h
#include <iostream>
#include <vector>
using std::cout; using std::endl; using std::vector;
int* doit(int length);
fast.cpp快速.cpp
#include "fast.h"
int* doit(int length) {
// Something really heavy
cout << "C++: doing it fast " << endl;
vector<int> WhyNot;
// Heavy stuff - like reading a big file and preprocessing it
for(int i=0; i<length; ++i)
WhyNot.push_back(i); // heavy stuff
cout << "C++: did it really fast" << endl;
return &WhyNot[0]; // or WhyNot.data()
}
faster.pyx更快.pyx
cimport numpy as np
import numpy as np
from libc.stdlib cimport free
from cpython cimport PyObject, Py_INCREF
np.import_array()
cdef extern from "fast.h":
int* doit(int length)
cdef class ArrayWrapper:
cdef void* data_ptr
cdef int size
cdef set_data(self, int size, void* data_ptr):
self.data_ptr = data_ptr
self.size = size
def __array__(self):
print ("Cython: __array__ called")
cdef np.npy_intp shape[1]
shape[0] = <np.npy_intp> self.size
ndarray = np.PyArray_SimpleNewFromData(1, shape,
np.NPY_INT, self.data_ptr)
print ("Cython: __array__ done")
return ndarray
def __dealloc__(self):
print("Cython: __dealloc__ called")
free(<void*>self.data_ptr)
print("Cython: __dealloc__ done")
def faster(length):
print("Cython: calling C++ function to do it")
cdef int *array = doit(length)
print("Cython: back from C++")
cdef np.ndarray ndarray
array_wrapper = ArrayWrapper()
array_wrapper.set_data(length, <void*> array)
print("Ctyhon: array wrapper set")
ndarray = np.array(array_wrapper, copy=False)
ndarray.base = <PyObject*> array_wrapper
Py_INCREF(array_wrapper)
print("Cython: all done - returning")
return ndarray
setup.py安装程序.py
from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext
import numpy
ext_modules = [Extension(
"faster",
["faster.pyx", "fast.cpp"],
language='c++',
extra_compile_args=["-std=c++11"],
extra_link_args=["-std=c++11"]
)]
setup(
cmdclass = {'build_ext': build_ext},
ext_modules = ext_modules,
include_dirs=[numpy.get_include()]
)
If you build this with如果你用
python setup.py build_ext --inplace
and run Python 3.6 interpreter, if you enter the following you'd get seg fault after a couple of tries.并运行 Python 3.6 解释器,如果你输入以下内容,你会在几次尝试后遇到段错误。
>>> from faster import faster
>>> a = faster(1000000)
Cython: calling C++ function to do it
C++: doing it fast
C++: did it really fast
Cython: back from C++
Ctyhon: array wrapper set
Cython: __array__ called
Cython: __array__ done
Cython: all done - returning
>>> a = faster(1000000)
Cython: calling C++ function to do it
C++: doing it fast
C++: did it really fast
Cython: back from C++
Ctyhon: array wrapper set
Cython: __array__ called
Cython: __array__ done
Cython: all done - returning
Cython: __dealloc__ called
Segmentation fault (core dumped)
Couple of things to note:需要注意的几件事:
faster(1000000)
and put the result into something other than variable a
this would work.如果您调用faster(1000000)
并将结果放入variable a
以外的其他内容中,这将起作用。 If you enter smaller number like faster(10)
you'd get a more detailed info like:如果您输入较小的数字,例如faster(10)
,您将获得更详细的信息,例如:
Cython: calling C++ function to do it
C++: doing it fast
C++: did it really fast
Cython: back from C++
Ctyhon: array wrapper set
Cython: __array__ called
Cython: __array__ done
Cython: all done - returning
Cython: __dealloc__ called <--- Perhaps this happened too early or late?
*** Error in 'python': double free or corruption (fasttop): 0x0000000001365570 ***
======= Backtrace: =========
More info here ....
It's really puzzling that why this doesn't happen with arrays?令人费解的是,为什么数组不会发生这种情况? No matter what!无论!
I make use of vectors a lot and would love to be able to use them in these scenarios.我经常使用矢量,并且希望能够在这些场景中使用它们。
I think @FlorianWeimer's answer provides a decent solution (allocate a vector
and pass that into your C++ function) but it should be possible to return a vector from doit
and avoid copies by using the move constructor.我认为@FlorianWeimer 的回答提供了一个不错的解决方案(分配一个vector
并将其传递给您的 C++ 函数)但是应该可以从doit
返回一个向量并通过使用移动构造函数避免复制。
from libcpp.vector cimport vector
cdef extern from "<utility>" namespace "std" nogil:
T move[T](T) # don't worry that this doesn't quite match the c++ signature
cdef extern from "fast.h":
vector[int] doit(int length)
# define ArrayWrapper as holding in a vector
cdef class ArrayWrapper:
cdef vector[int] vec
cdef Py_ssize_t shape[1]
cdef Py_ssize_t strides[1]
# constructor and destructor are fairly unimportant now since
# vec will be destroyed automatically.
cdef set_data(self, vector[int]& data):
self.vec = move(data)
# @ead suggests `self.vec.swap(data)` instead
# to avoid having to wrap move
# now implement the buffer protocol for the class
# which makes it generally useful to anything that expects an array
def __getbuffer__(self, Py_buffer *buffer, int flags):
# relevant documentation http://cython.readthedocs.io/en/latest/src/userguide/buffer.html#a-matrix-class
cdef Py_ssize_t itemsize = sizeof(self.vec[0])
self.shape[0] = self.vec.size()
self.strides[0] = sizeof(int)
buffer.buf = <char *>&(self.vec[0])
buffer.format = 'i'
buffer.internal = NULL
buffer.itemsize = itemsize
buffer.len = self.v.size() * itemsize # product(shape) * itemsize
buffer.ndim = 1
buffer.obj = self
buffer.readonly = 0
buffer.shape = self.shape
buffer.strides = self.strides
buffer.suboffsets = NULL
You should then be able to use it as:然后您应该可以将其用作:
cdef vector[int] array = doit(length)
cdef ArrayWrapper w
w.set_data(array) # "array" itself is invalid from here on
numpy_array = np.asarray(w)
Edit: Cython isn't hugely good with C++ templates - it insists on writing std::move<vector<int>>(...)
rather than std::move(...)
then letting C++ deduce the types.编辑: Cython 对于 C++ 模板不是很好——它坚持编写std::move<vector<int>>(...)
而不是std::move(...)
然后让 C++ 推导类型。 This sometimes causes problems with std::move
.这有时会导致std::move
出现问题。 If you're having issues with it then the best solution is usually to tell Cython about only the overloads you want:如果您遇到问题,那么最好的解决方案通常是只告诉 Cython 您想要的重载:
cdef extern from "<utility>" namespace "std" nogil:
vector[int] move(vector[int])
When you return from doit
, the WhyNot
object goes out of scope, and the array elements are deallocated.当您从doit
返回时, WhyNot
对象超出范围,并且数组元素被释放。 This means that &WhyNot[0]
is no longer a valid pointer.这意味着&WhyNot[0]
不再是一个有效的指针。 You need to store the WhyNot
object somewhere else, probably in a place provided by the caller.您需要将WhyNot
对象存储在其他地方,可能是在调用者提供的地方。
One way to do this is to split doit
into three functions, doit_allocate
which allocates the vector and returns a pointer to it, doit
as before (but with an argument which receives a pointer to the preallocated vector , and
doit_free` which deallocates the vector.一种方法是将doit
拆分为三个函数, doit_allocate
分配向量并返回指向它的指针, doit
和以前一样(但有一个参数接收指向预分配向量的指针, and
doit_free 释放向量。
Something like this:像这样:
vector<int> *
doit_allocate()
{
return new vector<int>;
}
int *
doit(vector<int> *WhyNot, int length)
{
// Something really heavy
cout << "C++: doing it fast " << endl;
// Heavy stuff - like reading a big file and preprocessing it
for(int i=0; i<length; ++i)
WhyNot->push_back(i); // heavy stuff
cout << "C++: did it really fast" << endl;
return WhyNot->front();
}
void
doit_free(vector<int> *WhyNot)
{
delete WhyNot;
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.