Return a 2D Cython pointer to Python array

Question

I am currently passing from Cython to C the following pointer of a pointer:

    #convert the input Python 2D array to a memory view
    cdef double[:,:] a_cython= np.asarray(a,order="C")
    #define a pointer of a pointer with dimensions of a
    cdef double** point_to_a = <double **>malloc(N * sizeof(double*))
    #initialize the pointer
    if not point_to_a: raise MemoryError
    #try:
    for i in range(N):
        point_to_a[i] = &a_cython[i, 0]

    #pass this double pointer to a C function
    logistic_sigmoid(&point_to_a[0], N,M)

where a is a numpy array, whose dimensions are N x M , point_to_a is a Cython pointer of a pointer which is referring to Cython memoryview a_cython . Since the input a from Python is 2 dimensional array, I thought this was the best approach to pass the info directly to C. The passage goes smoothly and the computation is done correctly. However, I am trying now to re-convert back point_to_a to a numpy array, but I am struggling a bit.

I am considering various solutions. I would like to explore if it's possible to keep a N dimensional array throughout the entire process, thus I was experimenting with this approach in Cython:

    #define a integer array for dimensions
    cdef np.npy_intp dims[2]
    dims[0]=  N
    dims[1] = M
    #create a new memory view and PyArray_SimpleNewFromData to deal with the pointer
    cdef np.ndarray[double, ndim=2] new_a =  np.PyArray_SimpleNewFromData(2, &dims[0], np.NPY_DOUBLE, point_to_a)

however, when I am converting new_a to a np.array as array = np.asarray(new_a) I have an array with 0s only. Do you have any ideas?

Thanks very much

Answer 1

As soon as you use int** (or similar) your data is in so-called indirect memory layout. Cython's typed memory views support indirect memory layout (see for example Cython: understanding a typed memoryview with a indirect_contignuous memory layout ), however there are not so many classes implementing this interface.

Numpy's ndarrays do not implement indirect memory layout - they only support direct memory layouts (eg pointer of type int* and not int** ), so passing an int** to a numpy array will do no good.

The good thing is, that because you share the memory with a_cython , the values were already updated in-place. You can get the underlying numpy array by returning the base -object of the typed memory view, ie

return a_cython.base # returns 2d-numpy array.

there is no need to copy memory at all!

---

There are however some issues with memory management (eg you need to free point_to_a ).

This is maybe an overkill in your case, but I use the opportunity to shamelessly plug-in a library of mine indirect_buffer : Because alternatives for indirect memory layout buffers are scarce and from time to time one needs one, I've create one to avoid writing always the same code.

With indirect_buffer your function could look like following:

%%cython
#just an example for a c-function
cdef extern from *:
    """
    void fillit(int** ptr, int N, int M){
       int cnt=0;
       for(int i=0;i<N;i++){
          for(int j=0;j<M;j++){
            ptr[i][j]=cnt++;
          }
       }
    }
    """
    void fillit(int** ptr, int N, int M)

from indirect_buffer.buffer_impl cimport IndirectMemory2D
def py_fillit(a):
    #create collection, it is a view of a
    indirect_view=IndirectMemory2D.cy_view_from_rows(a, readonly=False)
    fillit(<int**>indirect_view.ptr, indirect_view.shape[0], indirect_view.shape[1])
    # values are updated directly in a

which now can be used, for example:

import numpy as np
a=np.zeros((3,4), dtype=np.int32)
py_fillit(a)
print(a)
# prints as expected:
#  array([[ 0,  1,  2,  3],
#         [ 4,  5,  6,  7],
#         [ 8,  9, 10, 11]])

The above version does a lot of things right: memory management, locking of buffers and so on.