了解使用Cython扩展类型与Python类的好处

Question

I am very new to Cython and after skimming through the Cython documentation I came across Cython Extention types . I am wondering what benefit does it give over normal Python Classes? I was trying to convert my previous Python class which had lists in its data members to a Cython extension type but it seems like I cannot declare lists as data members in extension type. Only thing that I can do is convert a Python class which uses data-members of primitive C datatypes to Cython extension type.

I have a segment of my code in Python class which I need to optimize using Cython. How can I Cythonize functions related to that segment without declaring the class as a Cython extension type? (Basically I want to declare only certain functions as cdef, not all.)

Answer 1

I have not used those cdef ed classes when writing code in Cython, so I'll leave that part on benefits of cdef ed types for someone else to answer.

On the point of lists, while you can still use Python lists as normal, you will need to copy each member explicitly into a C array to fully take advantage of the speed-ups from writing in C. However, if the data is stored in a NumPy array, you can just store a pointer to the start of the array (making sure that the array is C-contiguous) instead. The full equivalence table of NumPy and C types can be found here: https://github.com/cython/cython/blob/master/Cython/Includes/numpy/__init__.pxd

As for cythonising class methods, since most Python code can work as is without modification in Cython, you can define your class in Cython using class SomeClass: as normal. Since cdef functions can only be called from within Cython, you would likely want to define cythonised methods outside the class (preferably typed to improve performance). Inside the class, you can use the regular def (which can be called from Python) to call their cythonised counterpart.

For larger mostly Python classes which you would not like to move into Cython, you can use a similar method but only have def ed functions in Cython calling the cdef ed version. You then call the Cython functions from Python as you would normally do when importing modules.

For data structures that only needs to reside in C and not interact (much) with Python, you may also want to consider using PyCapsule to store them as your class attributes.

Edit:

From reading the comment under chrisb 's answer, I gather that you want to have a 2D array with variable row length. Before you dive straight into implementing the exact same data structure in C, it is worth noting that C is not Python. It will not automatically manage the length of lists for you, instead you will have to manage the memory yourself (see the example in the first link). While this is the standard way of dynamically allocating memory, programmers new to C (and Cython) tend not to want to touch " malloc and friends" since pointers will be flying around. On top of that, C arrays generally do not have mixed data type, eg you cannot have both numbers and strings in the same array (if you really need to, there is a way to do that).

In light of this, you may wish to rethink your data structure. For example, you could consider way to represent the data with arrays of constant length. If your array has a maximum width, you may be able to exchange memory with ease of programming by using a NumPy array.

If you are happy to give manual memory management a go, here is a simple example of allocating memory for a 2D array of int s:

from cpython.mem cimport PyMem_Malloc, PyMem_Realloc, PyMem_Free

cdef int **generate_2D_array(int rows, int columns):
    cdef int row
    cdef int **parent = <int **>PyMem_Malloc(rows * sizeof(int*))
    if not parent:
        raise MemoryError()
    for row in range(rows):
        parent[row] = <int *>PyMem_Malloc(columns * sizeof(int))
        if not parent[row]:
            raise MemoryError()
    return parent

To change the length of a row, you can use:

cdef void resize_row(int *row_pointer, int new_size):
    PyMem_Realloc(row_pointer, new_size)

When you are done with the data, remember to deallocate the memory using PyMem_Free in a similar fashion to allocation with PyMem_Malloc . The rule of thumb is: for every PyMem_Malloc you use, free the memory using exactly one PyMem_Free , no more, no less. And finally, just a word of warning, failure to use these appropriately may cause segmentation faults, memory leaks or undefined behaviour.

Answer 2

For python-heavy operations, a cdef class will still have some performance benefits, due to optimized field access and python overhead removal. Simple example:

class PyThing:
    def __init__(self, data):
        self.data = data
    def calc(self):
        ans = 0
        for v in self.data:
            ans += v
        return ans


py = PyThing(list(range(100000)))

%timeit py.calc()
4.07 ms ± 29.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)


import cython
%load_ext cython

%%cython
cdef class CyThing:
    cdef list data
    def __init__(self, data):
        self.data = data
    def calc(self):
        ans = 0
        for v in self.data:
            ans += v
        return ans

cy = CyThing(list(range(100000)))

%timeit cy.calc()
2.8 ms ± 123 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

This is at the cost of simplicity / debug-ability, so may or may not be worth it. A better usecase is when you have some c-level values you want to store and use, there the performance benefits can be much more significant.