[英]Including and distributing third party libraries with a Python C extension
I'm building a C Python extension which makes use of a "third party" library— in this case, one that I've built using a separate build process and toolchain.我正在构建一个使用“第三方”库的 C Python 扩展——在本例中,我使用单独的构建过程和工具链构建了一个。 Call this library
libplumbus.dylib
.调用这个库
libplumbus.dylib
。
Directory structure would be:目录结构为:
grumbo/
include/
plumbus.h
lib/
libplumbus.so
grumbo.c
setup.py
My setup.py
looks approximately like:我的
setup.py
看起来大约像:
from setuptools import Extension, setup
native_module = Extension(
'grumbo',
define_macros = [('MAJOR_VERSION', '1'),
('MINOR_VERSION', '0')],
sources = ['grumbo.c'],
include_dirs = ['include'],
libraries = ['plumbus'],
library_dirs = ['lib'])
setup(
name = 'grumbo',
version = '1.0',
ext_modules = [native_module] )
Since libplumbus is an external library, when I run import grumbo
I get:由于 libplumbus 是一个外部库,当我运行
import grumbo
我得到:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: dlopen(/path/to/grumbo/grumbo.cpython-37m-darwin.so, 2): Library not loaded: lib/libplumbus.dylib
Referenced from: /path/to/grumbo/grumbo.cpython-37m-darwin.so
Reason: image not found
What's the simplest way to set things up so that libplumbus
is included with the distribution and properly loaded when grumbo
is imported?什么是最简单的设置方法,以便
libplumbus
包含在发行版中并在导入grumbo
时正确加载? (Note that this should work with a virtualenv). (请注意,这应该适用于 virtualenv)。
I have tried adding lib/libplumbus.dylib
to package_data
, but this doesn't work, even if I add -Wl,-rpath,@loader_path/grumbo/lib
to the Extension's extra_link_args
.我曾尝试将
lib/libplumbus.dylib
添加到package_data
,但这不起作用,即使我将-Wl,-rpath,@loader_path/grumbo/lib
到扩展的extra_link_args
。
The goal of this post is to have a setup.py
which would create a source distribution.这篇文章的目标是创建一个
setup.py
来创建一个源代码分发。 That means after running这意味着运行后
python setup.py sdist
the resulting dist/grumbo-1.0.tar.gz
could be used for installation via生成的
dist/grumbo-1.0.tar.gz
可用于安装
pip install grumbo-1.0.tar.gz
We will start for a setup.py
for Linux/MacOS, but then tweak to make it work for Windows as well.我们将从 Linux/MacOS 的
setup.py
开始,然后进行调整以使其也适用于 Windows。
The first step is to get the additional data (includes/library) into the distribution.第一步是将附加数据(包含/库)放入分发中。 I'm not sure it is really impossible to add data for a module, but
setuptools
offers functionality to add data for packages, so let's make a package from your module (which is probably a good idea anyway).我不确定是否真的不可能为模块添加数据,但是
setuptools
提供了为包添加数据的功能,所以让我们从您的模块中创建一个包(无论如何这可能是一个好主意)。
The new structure of package grumbo
looks as follows:包
grumbo
的新结构如下所示:
src/
grumbo/
__init__.py # empty
grumbo.c
include/
plumbus.h
lib/
libplumbus.so
setup.py
and changed setup.py
:并更改
setup.py
:
from setuptools import setup, Extension, find_packages
native_module = Extension(
name='grumbo.grumbo',
sources = ["src/grumbo/grumbo.c"],
)
kwargs = {
'name' : 'grumbo',
'version' : '1.0',
'ext_modules' : [native_module],
'packages':find_packages(where='src'),
'package_dir':{"": "src"},
}
setup(**kwargs)
It doesn't do much yet, but at least our package can be found by setuptools
.它还没有做太多事情,但至少我们的包可以通过
setuptools
找到。 The build fails, because the includes are missing.构建失败,因为缺少包含。
Now let's add the needed includes from the include
-folder to the distribution via package-data
:现在让我们通过
package-data
将include
夹中所需的包含添加到发行版中:
...
kwargs = {
...,
'package_data' : { 'grumbo': ['include/*.h']},
}
...
With that our include-files are copied to the source distribution.这样我们的包含文件就被复制到源代码分发版中。 However because it will be build "somewhere" we don't know yet, adding
include_dirs = ['include']
to the Extension
definition just doesn't cut it.然而,因为它将在我们还不知道的“某处”构建,将
include_dirs = ['include']
到Extension
定义中并不能削减它。
There must be a better way (and less brittle) to find the right include path, but that is what I came up with:必须有更好的方法(并且不那么脆弱)来找到正确的包含路径,但这就是我想出的:
...
import os
import sys
import sysconfig
def path_to_build_folder():
"""Returns the name of a distutils build directory"""
f = "{dirname}.{platform}-{version[0]}.{version[1]}"
dir_name = f.format(dirname='lib',
platform=sysconfig.get_platform(),
version=sys.version_info)
return os.path.join('build', dir_name, 'grumbo')
native_module = Extension(
...,
include_dirs = [os.path.join(path_to_build_folder(),'include')],
)
...
Now, the extension is built, but cannot be yet loaded because it is not linked against shared-object libplumbus.so
and thus some symbols are unresolved.现在,扩展已构建,但尚未加载,因为它未链接到共享对象
libplumbus.so
,因此某些符号未解析。
Similar to the header files, we can add our library to the distribution:与头文件类似,我们可以将我们的库添加到发行版中:
kwargs = {
...,
'package_data' : { 'grumbo': ['include/*.h', 'lib/*.so']},
}
...
and add the right lib-path for the linker:并为链接器添加正确的库路径:
...
native_module = Extension(
...
libraries = ['plumbus'],
library_dirs = [os.path.join(path_to_build_folder(), 'lib')],
)
...
Now, we are almost there:现在,我们快到了:
site-packages/grumbo/
site-packages/grumbo/
libplumbus.so
as can be seen with help of ldd
libplumbus.so
可以在ldd
帮助下看到libplumbus.so
is put into site-packages/grumbo/lib
libplumbus.so
被放入site-packages/grumbo/lib
However, we still cannot import the extension, as import grumbo.grumbo
leads to但是,我们仍然无法导入扩展名,因为
import grumbo.grumbo
导致
ImportError: libplumbus.so: cannot open shared object file: No such file or directory
导入错误:libplumbus.so:无法打开共享对象文件:没有这样的文件或目录
because the loader cannot find the needed shared object which resides in the folder .\\lib
relative to our extension.因为加载程序找不到驻留在文件夹
.\\lib
相对于我们的扩展所需的共享对象。 We could use rpath
to "help" the loader:我们可以使用
rpath
来“帮助”加载器:
...
native_module = Extension(
...
extra_link_args = ["-Wl,-rpath=$ORIGIN/lib/."],
)
...
And now we are done:现在我们完成了:
>>> import grumbo.grumbo
# works!
Also building and installing a wheel should work:构建和安装轮子也应该有效:
python setup.py bdist_wheel
and then:进而:
pip install grumbo-1.0-xxxx.whl
The first mile stone is achieved.实现了第一个里程碑。 Now we extend it, so it works other platforms as well.
现在我们扩展它,所以它也适用于其他平台。
Same source distribution for Linux and Macos: Linux 和 Macos 的相同源代码分发:
To be able to install the same source distribution on Linux and MacOS, both versions of the shared library (for Linux and MacOS) must be present.为了能够在 Linux 和 MacOS 上安装相同的源发行版,共享库的两个版本(适用于 Linux 和 MacOS)都必须存在。 An option is to add a suffix to the names of shared objects: eg having
libplumbus.linux.so
and libplumbis.macos.so
.一种选择是为共享对象的名称添加后缀:例如,具有
libplumbus.linux.so
和libplumbis.macos.so
。 The right shared object can be picked in the setup.py
depending on the platform:可以根据平台在
setup.py
选择正确的共享对象:
...
import platform
def pick_library():
my_system = platform.system()
if my_system == 'Linux':
return "plumbus.linux"
if my_system == 'Darwin':
return "plumbus.macos"
if my_system == 'Windows':
return "plumbus"
raise ValueError("Unknown platform: " + my_system)
native_module = Extension(
...
libraries = [pick_library()],
...
)
Tweaking for Windows:调整 Windows:
On Windows, dynamic libraries are dlls and not shared objects, so there are some differences that need to be taken into account:在 Windows 上,动态库是 dll 而不是共享对象,因此需要考虑一些差异:
plumbus.lib
-file, which we need to put into the lib
-subfolder.plumbus.lib
文件,我们需要将其放入lib
夹中。plumbus.dll
-file.plumbus.dll
文件。rpath
, thus we need to put the dll right next to the extension, so it can be found (see also this SO-post for more details). rpath
概念,因此我们需要将 dll 放在扩展名旁边,以便可以找到它(有关更多详细信息,另请参阅此SO-post )。 That means the folder structure should be as follows:这意味着文件夹结构应如下所示:
src/
grumbo/
__init__.py
grumbo.c
plumbus.dll # needed for Windows
include/
plumbus.h
lib/
libplumbus.linux.so # needed on Linux
libplumbus.macos.so # needed on Macos
plumbus.lib # needed on Windows
setup.py
There are also some changes in the setup.py
. setup.py
中也有一些变化。 First, extending the package_data
so dll
and lib
are picked up:首先,扩展
package_data
以便获取dll
和lib
:
...
kwargs = {
...
'package_data' : { 'grumbo': ['include/*.h', 'lib/*.so',
'lib/*.lib', '*.dll', # for windows
]},
}
...
Second, rpath
can only be used on Linux/MacOS, thus:其次,
rpath
只能在 Linux/MacOS 上使用,因此:
def get_extra_link_args():
if platform.system() == 'Windows':
return []
else:
return ["-Wl,-rpath=$ORIGIN/lib/."]
native_module = Extension(
...
extra_link_args = get_extra_link_args(),
)
That it!那个!
The complete setup file (you might want to add macro-definition or similar, which I've skipped):完整的安装文件(你可能想添加宏定义或类似的,我已经跳过了):
from setuptools import setup, Extension, find_packages
import os
import sys
import sysconfig
def path_to_build_folder():
"""Returns the name of a distutils build directory"""
f = "{dirname}.{platform}-{version[0]}.{version[1]}"
dir_name = f.format(dirname='lib',
platform=sysconfig.get_platform(),
version=sys.version_info)
return os.path.join('build', dir_name, 'grumbo')
import platform
def pick_library():
my_system = platform.system()
if my_system == 'Linux':
return "plumbus.linux"
if my_system == 'Darwin':
return "plumbus.macos"
if my_system == 'Windows':
return "plumbus"
raise ValueError("Unknown platform: " + my_system)
def get_extra_link_args():
if platform.system() == 'Windows':
return []
else:
return ["-Wl,-rpath=$ORIGIN/lib/."]
native_module = Extension(
name='grumbo.grumbo',
sources = ["src/grumbo/grumbo.c"],
include_dirs = [os.path.join(path_to_build_folder(),'include')],
libraries = [pick_library()],
library_dirs = [os.path.join(path_to_build_folder(), 'lib')],
extra_link_args = get_extra_link_args(),
)
kwargs = {
'name' : 'grumbo',
'version' : '1.0',
'ext_modules' : [native_module],
'packages':find_packages(where='src'),
'package_dir':{"": "src"},
'package_data' : { 'grumbo': ['include/*.h', 'lib/*.so',
'lib/*.lib', '*.dll', # for windows
]},
}
setup(**kwargs)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.