[英]Python ctypes 'c_char_p' memory leak
I'm developing a Python library for cryptography.我正在开发一个用于密码学的 Python 库。 I wanted to optimize my library by writing the main classes in C++ with GMP.
我想通过使用 GMP 用 C++ 编写主要类来优化我的库。 I wrote my C++ classes and I wrote the
extern
methods to use the main arithmetic operations: addition, subtraction, etc... These methods returns the results as char* to avoid cast problems.我编写了我的 C++ 类并编写了
extern
方法来使用主要的算术运算:加法、减法等......这些方法将结果返回为 char* 以避免转换问题。 I built the DLL of my library and I declared the methods in a Python wrapper with ctypes.我构建了库的 DLL,并在带有 ctypes 的 Python 包装器中声明了方法。 I noticed that after each arithmetic operation with huge numbers the memory grew exponentially.
我注意到在每次有大量数字的算术运算之后,内存呈指数增长。 I was looking for problems in my C++ implementation, but there were no problems thanks to the C++ garbage collector.
我一直在寻找我的 C++ 实现中的问题,但由于 C++ 垃圾收集器,没有问题。 I was looking for a possible solution, so I discovered that I had to implement a C++ method to free up memory of the string created by the DLL.
我正在寻找一个可能的解决方案,所以我发现我必须实现一个 C++ 方法来释放由 DLL 创建的字符串的内存。 So I wrote this simple method:
所以我写了这个简单的方法:
extern "C" {
__declspec(dllexport) void free_memory(char * n)
{
free(n);
}
...
}
I implemented this code in the Python wrapper to free up the memory allocated by the DLL:我在 Python 包装器中实现了此代码以释放 DLL 分配的内存:
import os
import ctypes
DIR_PATH = os.path.dirname(os.path.realpath(__file__))
NUMERIC = ctypes.CDLL(DIR_PATH + "/numeric.dll")
...
NUMERIC.free_memory.argtypes = [ctypes.c_void_p]
NUMERIC.free_memory.restype = None
def void_cast(n):
a = ctypes.cast(n, ctypes.c_char_p)
res = ctypes.c_char_p(a.value)
NUMERIC.free_memory(a)
return res
So with res = ctypes.c_char_p (a.value)
I create a new variable that no longer points to a
.因此,与
res = ctypes.c_char_p (a.value)
创建一个新的变量,它不再指向a
。 This way I correctly delete a
using the DLL method, but I still have memory leak problems.这样我使用DLL方法正确删除
a
,但我仍然有内存泄漏问题。 It is as if the Python garbage collector does not free up correctly the memory of strings of type c_char_p
.就好像 Python 垃圾收集器没有正确
c_char_p
类型的字符串的内存c_char_p
。 In the previous implementation I used only Python and the gmpy2
library, so all the numbers were converted to mpz
or mpq
.在之前的实现中,我只使用了 Python 和
gmpy2
库,因此所有数字都转换为mpz
或mpq
。 I tested the memory consumption using the memory_profiler
package.我使用
memory_profiler
包测试了内存消耗。 I created 40 objects of type projective point, defined on an elliptical curve, and I calculated the products i*P
, with i
from 1 to 40. With gmpy2
about 70MB in total were used.我创建了 40 个投影点类型的对象,定义在椭圆曲线上,我计算了
i*P
的乘积, i
从 1 到gmpy2
总共使用了大约 70MB。 Instead, using ctypes with the classes in C++ the consumption of the memory rose to 1.5GB.相反,将 ctypes 与 C++ 中的类一起使用,内存消耗上升到 1.5GB。 It's obvious that there is something wrong, especially when only the base classes that deal with arithmetic operations change.
很明显有问题,特别是当只有处理算术运算的基类发生变化时。 How can I properly free the memory without having memory leak problems?
如何在没有内存泄漏问题的情况下正确释放内存? I put an example of
extern
method for calculating an arithmetic operation, but I have already checked that the problem lies only in correctly freeing the memory via the free_memory
function and reassigning the string so that the garbage collector of Python will free the string when needed.我举了一个用于计算算术运算的
extern
方法的例子,但我已经检查过问题仅在于通过free_memory
函数正确释放内存并重新分配字符串,以便 Python 的垃圾收集器在需要时释放字符串。
extern "C" {
__declspec(dllexport) const char* rat_add(const char * n, const char * m)
{
return (RationalNum(n) + RationalNum(m)).getValue();
}
}
Thanks in advance and have a nice day.提前致谢,祝您有美好的一天。
PS: Clearly in C++ I correctly implemented the destructor method to free up the space of the mpz_t
and mpq_t
objects created. PS:显然在 C++ 中,我正确地实现了析构函数方法来释放创建的
mpz_t
和mpq_t
对象的空间。
The problem is in this line:问题出在这一行:
res = ctypes.c_char_p(a.value)
This creates a copy of a.value
and sets res
to a c_char_p
that points to the copy.这将创建的副本
a.value
,并设置res
的c_char_p
指向副本。 However, Python does not do memory management for ctypes
pointers, so the copy will be leaked!但是Python不会对
ctypes
指针做内存管理,所以副本会泄露!
The leak should be fixed if you replace above line with:如果将以上行替换为以下内容,则应修复泄漏:
res = bytes(memoryview(a.value))
This also creates a copy, but res
will be a real Python object.这也会创建一个副本,但
res
将是一个真正的 Python 对象。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.