[英]Cross-compile extension on Linux for Windows
I've managed to build some DLLs on Linux that are necessary for my Python extension using MinGW.我已经设法在 Linux 上构建了一些 DLL,这些 DLL 是使用 MinGW 的 Python 扩展所必需的。 Something along these lines:
沿着这些路线的东西:
from setuptools.command.build_py import build_py
class BuildGo(build_py):
def run(self):
if # need to build windows binaries
self.build_win()
build_py.run(self)
def build_win(self):
if # compilers and toolchain available
try:
# builds extra libraries necessary for this extension
except subprocess.CalledProcessError as e:
print(e.stderr)
raise
try:
result = subprocess.check_output([
'x86_64-w64-mingw32-gcc-win32',
'-shared',
'-pthread',
'-o',
EXTRA_DLL,
FAKE_WIN_BINDINGS,
ARCHIVE_GENERATED_IN_PREVIOUS_STEP,
'-lwinmm',
'-lntdll',
'-lws2_32',
])
print(result)
except subprocess.CalledProcessError as e:
print(e.stderr)
raise
I was now hoping I could avoid extending build_ext
in the same painful way to get it to cross-compile Cython code for Windows... I looked into the abyss of "elegant interplay of setuptools
, distutils
and cython
", and before the abyss has a chance to look back into me... Isn't there a way to just specify some flag... like a name of compiler and Python binary for desired platform and... it would just do it?我现在希望我能避免延长
build_ext
在相同的痛苦的方式得到它的交叉编译的Windows用Cython代码...我看着的“优雅相互深渊setuptools
, distutils
和cython
”,和之前的深渊有机会回顾一下我......难道没有办法只指定一些标志......比如所需平台的编译器和Python二进制文件的名称......它会这样做吗?
I've read this article: http://whatschrisdoing.com/blog/2009/10/16/cross-compiling-python-extensions/ - it's almost 10 years old.我读过这篇文章: http : //whatschrisdoing.com/blog/2009/10/16/cross-compiling-python-extensions/ - 它已经快 10 岁了。 And it just made me want to cry... did anything change since it was written?
简直让我想哭……写完之后有什么变化吗? Or are these steps more or less what I'll have to do to compile for the platform other than the one I'm running on?
或者这些步骤是否或多或少是我为我正在运行的平台以外的平台编译的?
Or, is there an example project on the web which does it?或者,网络上是否有一个示例项目可以做到这一点?
My ultimate goal is to produce an egg
package which will contain both PE and ELF binaries in it and will install them in the correct location on either platform when installed by pip
or pipenv
.我的最终目标是生成一个包含 PE 和 ELF 二进制文件的
egg
包,并在通过pip
或pipenv
安装时将它们安装在pipenv
台上的正确位置。 It should compile on Linux (compiling it on MS Windows isn't necessary).它应该在 Linux 上编译(不需要在 MS Windows 上编译)。
I'm posting this as community wiki because it's a pretty unsatisfactory answer: it only tells you why it's very hard rather than offers really solutions.我将此作为社区维基发布,因为这是一个非常不令人满意的答案:它只告诉您为什么它非常困难,而不是提供真正的解决方案。
The official Python distributions on Windows are compiled with Microsoft Visual C (MSVC) , and when compiling a Python extension it's generally necessary to use the same version as the one that Python was compiled with. Windows 上的官方 Python 发行版是使用 Microsoft Visual C (MSVC)编译的,在编译 Python 扩展时,通常需要使用与编译 Python 时使用的版本相同的版本。 This shows you that an exact compiler match is pretty important.
这表明精确的编译器匹配非常重要。
It is possible to get versions of Python compiled with Mingw, and these would then be compatible with modules compiled with Mingw.有可能获得用 Mingw 编译的 Python 版本,然后这些版本将与用 Mingw 编译的模块兼容。 This could probably be made to work as a cross-compiler on Linux but the modules would only be useful to a very small subset of people that have this custom build of Python (so doesn't help create a useful distributable .egg file).
这可能可以用作 Linux 上的交叉编译器,但这些模块仅对拥有此自定义 Python 构建的一小部分人有用(因此无助于创建有用的可分发 .egg 文件)。
A reasonable effort has gone also into making a version of Mingw that can build compatible Python extensions on Windows: https://mingwpy.github.io/ (and I think also https://anaconda.org/msys2/m2w64-toolchain ).还做出了合理的努力,制作了一个可以在 Windows 上构建兼容 Python 扩展的 Mingw 版本: https ://mingwpy.github.io/(我认为也是https://anaconda.org/msys2/m2w64-toolchain ) . The main driver for this seems to be the lack of freely Fortran compiler for Windows that is compatible with MSVC, hence the ability to build Fortran modules is very useful.
这的主要驱动因素似乎是缺乏与 MSVC 兼容的适用于 Windows 的免费 Fortran 编译器,因此构建 Fortran 模块的能力非常有用。 The mingwpy toolchain worked pretty well in my experience, until Python 3.4 when the switch to a more recent version of MSVC brought a whole exciting new set of compatibility issues .
根据我的经验,mingwpy 工具链运行良好,直到 Python 3.4 时切换到更新版本的 MSVC 带来了一系列令人兴奋的新兼容性问题。
My feeling would be that any viable solution would probably be based around these mostly-working Mingw compilers for windows.我的感觉是,任何可行的解决方案都可能基于这些适用于 Windows 的主要工作的 Mingw 编译器。
According to https://docs.python.org/3/distutils/builtdist.html , distutils
only supports cross-compiling between win32
and win_amd64
as of this writing ( 3.7
).根据https://docs.python.org/3/distutils/builtdist.html ,截至撰写本文时,
distutils
仅支持win32
和win_amd64
之间的交叉编译( 3.7
)。
Moreover, building extensions with compilers other than the MSVC that Python is built with is not officially supported .此外, 官方不支持使用除 Python 构建的 MSVC 之外的编译器构建扩展。
It is theoretically possible by getting a Linux toolchain for win32/64
(including the necessary headers and link libraries), a set of the necessary Python for Windows binaries to link against, then forge compiler and linker paths and/or options in setup.py
-- though it will still be an unsupported setup.理论上可以通过获取用于
win32/64
的 Linux 工具链(包括必要的头文件和链接库)、一组用于链接 Windows 二进制文件的必要 Python,然后在setup.py
伪造编译器和链接器路径和/或选项-- 尽管它仍然是不受支持的设置。
So you'll be better off using a Windows VM or an online build service like AppVeyor.因此,您最好使用 Windows VM 或 AppVeyor 等在线构建服务。
I had the same issue once, but I just used a virtual machine to compile my most painfuly microsoft dependant programs.我曾经遇到过同样的问题,但我只是使用虚拟机来编译我最痛苦的微软依赖程序。
https://developer.microsoft.com/en-us/windows/downloads/virtual-machines https://developer.microsoft.com/en-us/windows/downloads/virtual-machines
If you don't have access to a windows machine or your programs uses very specific machiney like a fortran compiler optimized or some POSIX dependant stuff or newest features from VS redistributable versions, you better give a try to a virtual machine based compilation system.如果您无法访问 Windows 机器,或者您的程序使用非常特定的机器,例如优化的 fortran 编译器或某些 POSIX 相关的东西或来自 VS 可再发行版本的最新功能,您最好尝试基于虚拟机的编译系统。
Here is a proof of concept for cross compiling (Cython-) extensions for Windows on Linux.这是 Linux 上 Windows 交叉编译 (Cython-) 扩展的概念证明。
But first a word of warning: While possible, the workflow is not really supported (it starts with the fact that the only supported windows compiler is MSVC ), so it can be brocken with changes in future version.但首先要提醒一句:虽然可能,但并不真正支持工作流程(它开始于唯一支持的 Windows 编译器是MSVC ),因此它可能会因未来版本的更改而中断。 I use Python 3.7 for 64bit, things might be (slightly) different for other versions.
我将 Python 3.7 用于 64 位,其他版本的情况可能(略有)不同。
There might be legit scenarios for cross-compilation for Windows, but the python world seems to live quite good without, so probably cross-compilation is not the right direction in the most cases.对于 Windows 交叉编译可能有合法的场景,但是没有 Python 世界似乎还不错,所以在大多数情况下交叉编译可能不是正确的方向。
Preliminaries:预赛:
sudo apt-get install mingw-w64
) - the compiler for 64bit is x86_64-w64-mingw32-gcc
.sudo apt-get install mingw-w64
) - 64 位的编译器是x86_64-w64-mingw32-gcc
。 distutils
do no support mingw-w64
, so we will perform all steps manually. distutils
不支持mingw-w64
,因此我们将手动执行所有步骤。
1. C code generation 1. C 代码生成
Let's take the following simple Cython-extension foo.pyx
让我们以下面简单的 Cython-extension
foo.pyx
print("It is me!")
which can be transformed to C-code via:可以通过以下方式转换为 C 代码:
>>> cython -3 foo.pyx
which creates the foo.c
-file.它创建了
foo.c
文件。
2. Compilation 2.编译
The compilation step is:编译步骤是:
>>> x86_64-w64-mingw32-gcc -c foo.c -o foo.o -I <path_to_windows_includes> -DMS_WIN64 -O2 <other compile flags>
I guess one can be minimalistic and only use -O2
compile flag in most cases.我想可以是简约的,并且在大多数情况下只使用
-O2
编译标志。 It is however important to define MS_WIN64
-macro (eg via -DMS_WIN64
).然而,定义
MS_WIN64
-macro (例如通过-DMS_WIN64
)很重要。 In order to build for x64 on windows it must be set, but it works out of the box only for MSVC (defining _WIN64
could have slightly different outcomes):为了在 Windows 上为 x64 进行构建,必须对其进行设置,但它仅适用于 MSVC开箱即用(定义
_WIN64
可能会产生略有不同的结果):
#ifdef _WIN64
#define MS_WIN64
#endif
3. Linking 3. 链接
The linking command is:链接命令是:
>>> x86_64-w64-mingw32-gcc --shared foo.o -o foo.pyd -L <path_to_windows_dll> -lpython37
It is important, that the python-library ( python37
) should be the dll itself and not the lib
(see this SO-post ).重要的是,python 库(
python37
)应该是 dll 本身,而不是lib
(请参阅此SO-post )。
One probably should add the proper suffix to the resulting pyd-file, I use the old convention for simplicity here.人们可能应该为生成的 pyd 文件添加正确的后缀,为了简单起见,我在这里使用旧约定。
4. Running: 4.运行:
Copying pyd-file to windows and now:将 pyd 文件复制到 Windows,现在:
import foo
# prints "It is me!"
Done!完毕!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.