[英]How do I use the correct dll files to enable 3rd party C libraries in a Cython C extension?
I have a C function that involves decompressing data using zstd.我有一个涉及使用 zstd 解压缩数据的 C function。 I am attempting to call that function using Cython.我正在尝试使用 Cython 调用 function。
Using this page from the docs as a guide I can compile and run the code below with no problem. 使用文档中的这个页面作为指南,我可以毫无问题地编译和运行下面的代码。
(I don't actually use the zstd lib here) (我实际上并没有在这里使用 zstd 库)
// hello.c
#include <stdio.h>
#include <zstd.h>
int hello() {
printf("Hello, World!\n");
void *next_in = malloc(0);
void *next_out = malloc(0);
return 0;
}
# Hello.pyx
cdef extern from "hello.c":
int hello()
cpdef int callHello():
hello()
# hello_wrapper.setup.py
from setuptools import setup, Extension
from Cython.Build import cythonize
ext_modules = [
Extension(
"hello_wrapper",
["hello_wrapper.pyx"],
libraries=["zstd"],
library_dirs=["path/to/zstd/lib"],
include_dirs=['path/to/zstd/include'],
)
]
setup(
ext_modules = cythonize(ext_modules, gdb_debug=True)
)
Using the commands as follows I get the expected output:使用如下命令,我得到了预期的 output:
>py hello_wrapper.setup.py build_ext --inplace
>py
Python 3.8.3 (tags/v3.8.3:6f8c832, May 13 2020, 22:20:19) [MSC v.1925 32 bit (Intel)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import hello_wrapper
>>> hello_wrapper.callHello()
Hello, World!
0
However when I modify hello.c
to actually use the zstd library:但是,当我修改hello.c
以实际使用 zstd 库时:
// hello.c
#include <stdio.h>
#include <zstd.h>
int hello() {
printf("Hello, World!\n");
void *next_in = malloc(0);
void *next_out = malloc(0);
size_t const dSize = ZSTD_decompress(next_out, 0, next_in, 0); //the added line
return 0;
}
While hello_wrapper.setup.py
compiles fine, when I get to the import statement, I get the following error:虽然hello_wrapper.setup.py
编译得很好,但当我进入 import 语句时,我收到以下错误:
>>> import hello_wrapper
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: DLL load failed while importing hello_wrapper: The specified module could not be found.
From reading This SO article , I gather that this error means I'm not correctly pointing to or perhaps creating in the first place the required DLL files for zstd.lib to work its magic.通过阅读这篇 SO 文章,我了解到这个错误意味着我没有正确指向或者可能首先创建了 zstd.lib 所需的 DLL 文件来发挥它的魔力。 Is this correct?这个对吗? If so, how might I do that?如果是这样,我该怎么做? If not, what is the problem?如果不是,问题是什么?
We link our cython-extension against a windows-dll, that means:我们将 cython 扩展与 windows-dll 链接起来,这意味着:
*.lib
-file (ie zstd.lib
) is needed in "path/to/zstd/lib"
during the compile time *.lib
- 编译时需要在"path/to/zstd/lib"
中的文件(即zstd.lib
)*.dll
-file (ie zstd.dll
) is needed somewhere where Windows can find it when the module is imported. *.dll
文件(即zstd.dll
)在导入模块时 Windows 可以找到的地方需要。 Normally, Windows will not look in the "path/to/zstd/lib"
.通常, Windows 不会在"path/to/zstd/lib"
中查找。 And so we get a somewhat cryptic error message:所以我们得到了一个有点神秘的错误信息:
ImportError: DLL load failed: The specified module could not be found. ImportError: DLL 加载失败:找不到指定的模块。
Which doesn't mean there is something wrong with the module - it just happens to depend on a dll which cannot be found.这并不意味着该模块有问题 - 它只是碰巧依赖于无法找到的 dll。
While linux has -rpath
-option for the linker with which "path/to/zstd/lib"
could be passed (it can be added with runtime_library_dirs
-argument to Extension
), there is no such option on Windows.虽然 linux 具有-rpath
的 -rpath 选项,可以通过该选项传递"path/to/zstd/lib"
(它可以与runtime_library_dirs
-argument 添加到Extension
),但 ZABB4600699CEAA3DACB460069EB 上没有这样的选项。 The dll-search-algorithmus for Windows can be found here . Windows 的 dll 搜索算法可以在这里找到。 In a nutshell, dll is searched in (possible in another order as presented here)简而言之,搜索 dll(可能按此处提供的其他顺序)
C:\Windows\System32
)系统目录(例如C:\Windows\System32
)C:\Windows
) windows-目录(例如C:\Windows
)Putting the dll into system- or windows-directory doesn't sound too appealing, which leave us with the following options:将 dll 放入系统目录或 windows 目录听起来不太吸引人,这让我们有以下选择:
zstd.dll
next to the compiled extension (最简单的?)复制已编译扩展旁边的zstd.dll
PATH
-variable, eg set PATH="path/to/zstd/lib";%PATH%
将 zstd-path 添加到PATH
-变量中,例如set PATH="path/to/zstd/lib";%PATH%
Another option is somewhat more tricky: Given that另一种选择有点棘手: 鉴于
If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL. If a DLL with the same module name is already loaded in memory, the system checks only for redirection and a manifest before resolving to the loaded DLL, no matter which directory it is in. The system does not search for the DLL.
we can use ctypes
to "preload" the right dll, which will be used (without the need to search for it on the disc) when the wrapper-module is imported, ie:我们可以使用ctypes
来“预加载”正确的 dll,这将在导入 wrapper-module 时使用(无需在光盘上搜索),即:
import ctypes;
ctypes.CDLL("path/to/zstd/lib/zstd.dll"); # we preload with the full path
import hello_wrapper # works now!
The above applies if the extension is built and used on the same system (eg via build_ext --inplace
).如果扩展是在同一系统上构建和使用的(例如通过build_ext --inplace
),则上述适用。 installation/distribution is somewhat more cumbersome (this is covered by this SO-post ), one idea would be:安装/分发有点麻烦(这个SO-post涵盖了这一点),一个想法是:
*.h
-, *.lib
- and *.dll
-files into 'package_data' (it seems to happen automatically anyway)将*.h
-、 *.lib
- 和*.dll
-files 放入“package_data”(无论如何它似乎是自动发生的)library_path
(or programmatically the absolute path) can be set in the setup.py
so *.lib
is found by the linker.可以在setup.py
中设置正确的相对library_path
(或以编程方式绝对路径),因此 linker 可以找到*.lib
。*.pyd
-file in the installation. dll 将放在安装中已编译的*.pyd
文件旁边。 An example could be the following more or less minimal setup.py
, where everything (pyx-file, h-files, lib-file, dll-file) are put into a package/folder src/zstd
:一个示例可能是以下或多或少的最小setup.py
,其中所有内容(pyx-file、h-files、lib-file、dll-file)都放入包/文件夹src/zstd
:
from setuptools import setup, Extension, find_packages
from Cython.Build import cythonize
ext_modules = [
Extension(
"zstd.zstdwrapper",
["src/zstd/zstdwrapper.pyx"],
libraries=["zstd"],
library_dirs=["src/zstd"],
include_dirs=[], # set automatically to src/zstd during the build
)
]
print(find_packages(where='src'))
setup(
name = 'zstdwrapper',
ext_modules = cythonize(ext_modules),
packages = find_packages(where='src'),
package_dir = {"": "src"},
)
And now it can be installed with python setup.py install
or used to create eg a source-distribution via python setup.py sdist
which then can be installed via pip
.现在它可以使用python setup.py install
或用于通过python setup.py sdist
创建例如源分发,然后可以通过pip
安装。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.