python passing integer to c++ 64bit library: varying sizes

Question

I am loading a 64bit c++ library in a python soft (also 64bit). The first trouble I noticed is that some of the integers passed to the library through the following protocole were not of the right size:

Python side:

c_int = c_int64
polynomialFittingDLL.loadComputationFromPython.restype = None
polynomialFittingDLL.loadComputationFromPython.argtypes = [c_void_p,POINTER(c_int),POINTER(c_int),c_int,c_int,c_int,c_bool,POINTER(c_double),POINTER(c_double),POINTER(c_double)] 


self.polynomialFittingDLL.loadComputationFromPython(self.dll,
                              self.btuplesDllFlat.ctypes.data_as(POINTER(c_int)),
                              self.bterm.ctypes.data_as(POINTER(c_int)),
                              len(self.bterm),
                              sum(self.bterm),
                              len(self.rangeX),
                              baseType,
                              self.coefficients.ctypes.data_as(POINTER(c_double)),
                              self.rangeX.ctypes.data_as(POINTER(c_double)),
                              self.minX.ctypes.data_as(POINTER(c_double)))

C++ side:

#ifdef _MSC_VER
    typedef __int64 int64_t;
#else
    #include <stdint.h>
#endif

DLL_EXPORT void loadComputationFromPython(PolynomialCurve* curve,__int64 *_btuple,
                     int *_bterm, int _btermSize,
                     int _Nvalidbtuple,int _Nparams,bool _polynomial,
                     double *_coefficients,
                     double *_rangeX,double *_minX) 
{
curve->loadComputationFromPython(_btuple,_bterm,_btermSize,_Nvalidbtuple,_Nparams,_polynomial,_coefficients,_rangeX,_minX);
}

void PolynomialCurve::loadComputationFromPython(__int64 *_btuple,
                     int *_bterm, int _btermSize,
                     int _Nvalidbtuple,int _Nparams,bool _polynomial,
                     double *_coefficients,
                     double *_rangeX,double *_minX) 
{
btermSize = _btermSize;
Nparams = _Nparams;
btuple = new int*[btermSize];
for (int i=0;i<btermSize;++i) {
    btuple[i] = new int[Nparams];
    for (int j=0;j<Nparams;++j) {
        btuple[i][j] = _btuple[i*Nparams+j];
    }
}
bterm = _bterm;
NvalidBtuple = _Nvalidbtuple;
polynomial = _polynomial;
coefficients = _coefficients;
rangeX = _rangeX;
minX = _minX;
}

I had to use the __int64 type in order to have correct integer type for _btuple . But the other int pointers seems still to be having trouble dealing with the integer size...

My question is how can I be coherent inside the c++ code to have all the int attributes always the same bit size? which size I want to be coherent with the c_int size in python. Is the __int64 type adequate for my problem? can I define pointers of __int64? should I be similarly carefull with doubles?

Those problem appeared when I changed from a 32 bit compilation to a 64 bit. Thanks

Edit: what has further be observed is that:

from python to c++: the python integers are 64bytes integers and I have to specify in C++ that the received integers are 64Bytes (using the __int64 type)

from C++ to python: the int type in C++ is a 32bytes integers, and when I recieve integers in python I have to specify its size with the c_int32 type:

addrBtuples = self.polynomialFittingDLL.getBtuples(self.dll)
self.btuplesDllFlat = np.ctypeslib.as_array((c_int32 * self.btuplesSize).from_address(addrBtuples))

So, in order to be as much portable as possible: should I "cast" the python int to a 32 bytes one? or should I force the c++ library to work with __int64 types?

Answer 1

I'm not sure if I understand your question, but you should not "shadow" c_int with c_int64 : a c_int type is still 4 bytes on 64-bit

>>> import platform
>>> platform.architecture()
('64bit', 'WindowsPE')
>>> import ctypes
>>> ctypes.sizeof(ctypes.c_int)
4

The second argument should be a POINTER(ctypes.c_int64) :

# c_int = c_int64
polynomialFittingDLL.loadComputationFromPython.restype = None
polynomialFittingDLL.loadComputationFromPython.argtypes = [c_void_p,POINTER(ctypes.c_int64),POINTER(c_int),c_int,c_int,c_int,c_bool,POINTER(c_double),POINTER(c_double),POINTER(c_double)] 

Also, pointers keep the same size, whatever the type they points to.

Answer 2

The size of int and double from the C side is implementation dependent (the standard only specifies the minimum size). So, it could depend on the compiler. Even though, there is in most cases a consistency between what ctypes python library does and what the C++ compiler does in the same platform.

So, to solve this, you need to change the sizes from the python side in a coherent way to get the proper casting. Given the C function signature in your case, I think that the following signature in the python side could work (only changing the first to c_longlong):

polynomialFittingDLL.loadComputationFromPython.argtypes = [c_void_p,
                POINTER(c_longlong),
                POINTER(c_int),
                c_int,c_int,c_int,c_bool,
                POINTER(c_double),
                POINTER(c_double),POINTER(c_double)]

But if your C++ compiler is augmenting the sizes of ints you should try:

polynomialFittingDLL.loadComputationFromPython.argtypes = [c_void_p,
                POINTER(c_longlong),
                POINTER(c_long),
                c_long,c_long,c_long,c_bool,
                POINTER(c_double),
                POINTER(c_double),POINTER(c_double)]

and if doubles are also augmented:

polynomialFittingDLL.loadComputationFromPython.argtypes = [c_void_p,
                POINTER(c_longlong),
                POINTER(c_long),
                c_long,c_long,c_long,c_bool,
                POINTER(c_longdouble),
                POINTER(c_longdouble),POINTER(c_longdouble)]

See https://docs.python.org/2/library/ctypes.html for further information in all ctypes data types.

The point is that you need to check which is the casting that fits between ctypes and the compiled module.

I would recommend to test the border cases (long and integer maximum and minimum sizes, and the overflow) to see that everything is working properly.