[英]Compiling with Cython using OpenMP on macOS
I'm on macOS Mojave 10.14.6 and I'm trying to compile some required extensions modules in c and c++ from this repository with:我在 macOS Mojave 10.14.6 上,我正在尝试从这个存储库中用 c 和 c++ 编译一些必需的扩展模块:
python setup.py build_ext --inplace
which gives me the following error:这给了我以下错误:
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
/Users/user/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py:249: UserWarning:
!! WARNING !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++) is not compatible with the compiler Pytorch was
built with for this platform, which is clang++ on darwin. Please
use clang++ to to compile your extension. Alternatively, you may
compile PyTorch from source using g++, and then you can also use
g++ to compile your extension.
See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
with compiling PyTorch from source.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! WARNING !!
warnings.warn(WRONG_COMPILER_WARNING.format(
and further down:再往下:
clang: error: unknown argument: '-i'
clang: error: no such file or directory: 'sysroot'
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.
(this should be the relevant part, most recent tracebacks also show subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
and finally RuntimeError: Error compiling objects for extension
) (这应该是相关的部分,最近的回溯也显示subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
最后RuntimeError: Error compiling objects for extension
)
From what I understand and what I tried so far:根据我的理解和我到目前为止的尝试:
-fopenmp
in the setup file is one of the problems, seems like OpenMP is missing设置文件中的标志-fopenmp
是问题之一,似乎缺少 OpenMPbrew install llvm
and brew install libomp
因此我安装了brew install llvm
和brew install libomp
PATH=\\"/usr/local/bin:$PATH\\
to my .bash_profile
hoping that the newly installed compilers are now used将PATH=\\"/usr/local/bin:$PATH\\
到我的.bash_profile
希望现在可以使用新安装的编译器-Xpreprocessor
before -fopenmp
to the setup file according to here , so that extra_compile_args=['-Xpreprocessor', '-std=c99', '-O3', '-fopenmp']
加入该标志-Xpreprocessor
之前-fopenmp
根据到设置文件在这里,使extra_compile_args=['-Xpreprocessor', '-std=c99', '-O3', '-fopenmp']
CC=gcc python setup.py build_ext --inplace
, because of the warning about treating 'c' as 'c++' but since I'm trying to build c and c++ modules, I'm not sure what the variable should actually be set to我尝试了CC=gcc python setup.py build_ext --inplace
,因为关于将“c”视为“c++”的警告,但由于我正在尝试构建 c 和 c++ 模块,我不确定变量实际上应该是什么设置为CC=/usr/local/opt/llvm/bin/clang++ python setup.py build_ext --inplace
as suggested here我也尝试过CC=/usr/local/opt/llvm/bin/clang++ python setup.py build_ext --inplace
按照这里的建议-fopenmp
(which is not actually an option, because I need the parallel execution which is provided by OpenMP IIRC) results basically in the same error without the error about OpenMP尝试在没有-fopenmp
情况下构建它(这实际上不是一个选项,因为我需要 OpenMP IIRC 提供的并行执行)基本上会导致相同的错误,而没有关于 OpenMP 的错误CXXFLAGS
and CFLAGS
?我还需要设置CXXFLAGS
和CFLAGS
吗?I'm sorry for this confusing question, but having no prior knowledge about compilers and trying to solve this is like going down the rabbit hole.... I would be very grateful for any comment shedding light on this situation and I'm happy to provide any missing information!我很抱歉这个令人困惑的问题,但没有关于编译器的先验知识并试图解决这个问题就像掉进兔子洞......我将非常感谢任何关于这种情况的评论,我很高兴提供任何缺失的信息!
UPDATE:更新:
.bash_profile
and exporting export CC='gcc-10'; export CXX='clang++'
清理.bash_profile
并导出export CC='gcc-10'; export CXX='clang++'
export CC='gcc-10'; export CXX='clang++'
got rid of the Pytorch warning, but clang problem remains export CC='gcc-10'; export CXX='clang++'
摆脱了 Pytorch 警告,但 clang 问题仍然存在os.environ['CC'] = 'gcc-10'
and os.environ['CXX'] = 'clang++
in the setup.py
在setup.py
仍然保留os.environ['CC'] = 'gcc-10'
和os.environ['CXX'] = 'clang++
The complete error message is:完整的错误信息是:
(torch) [10:48:05] vanessamac: occupancy_networks $ python setup.py build_ext --inplace --verbose
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
building 'im2mesh.utils.libkdtree.pykdtree.kdtree' extension
Emitting ninja build file /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
[2/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _run_ninja_build
subprocess.run(
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/subprocess.py", line 512, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "setup.py", line 112, in <module>
setup(
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/__init__.py", line 165, in setup
return distutils.core.setup(**attrs)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/core.py", line 148, in setup
dist.run_commands()
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 966, in run_commands
self.run_command(cmd)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 985, in run_command
cmd_obj.run()
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 87, in run
_build_ext.run(self)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 340, in run
self.build_extensions()
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
build_ext.build_extensions(self)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
self._build_extensions_serial()
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
self.build_extension(ext)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
_build_ext.build_extension(self, ext)
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
objects = self.compiler.compile(sources,
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 469, in unix_wrap_ninja_compile
_write_ninja_file_and_compile_objects(
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1228, in _write_ninja_file_and_compile_objects
_run_ninja_build(
File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
raise RuntimeError(message)
RuntimeError: Error compiling objects for extension
Here are a few hints:这里有一些提示:
Use gcc instead of llvm or clang for painless openmp-support on macOS.在 macOS 上使用 gcc 而不是 llvm 或 clang 来获得无痛的 openmp 支持。 Note that apple's default gcc is just an alias for Apple clang as you'll see with gcc --version
.请注意,apple 的默认 gcc 只是 Apple clang 的别名,正如您将在gcc --version
看到的那样。 You can install the real gcc with homebrew: brew install gcc
.您可以使用自制软件安装真正的 gcc : brew install gcc
。
Then use export CC='gcc-10'
(the newest version should be gcc 10.x) inside the same terminal window to use homebrew's gcc temporarily as your C compiler.然后在同一个终端窗口中使用export CC='gcc-10'
(最新版本应该是 gcc 10.x)来临时使用 homebrew 的 gcc 作为你的 C 编译器。
There's no need to set CXXFLAGS
or CFLAGS
.无需设置CXXFLAGS
或CFLAGS
。 The required flags are set by distutils/setuptools inside the setup.py.所需的标志由 setup.py 中的 distutils/setuptools 设置。
You won't be able to compile dmc_cuda_module
on macOS 10.14.6.您将无法在 macOS dmc_cuda_module
上编译dmc_cuda_module
。 The latest macOS version nvidia offers cuda drivers for is 10.13.6.最新的 macOS 版本 nvidia 提供的 cuda 驱动程序为 10.13.6。 So you might uncomment this part of the setup.py and hope for the best you don't need this module...所以你可能会取消对 setup.py 的这一部分的注释,并希望你不需要这个模块......
Some of the Extensions inside the setup.py
aren't including the numpy headers while using the numpy C-API. setup.py
中的一些扩展在使用 numpy C-API 时不包括 numpy 标头。 On macOS it's necessary to include the numpy headers for each Extension, see this comment .在 macOS 上,有必要为每个扩展包含 numpy 标头,请参阅此注释。 So you have to add include_dirs=[numpy_include_dir]
to those Extensions.因此,您必须将include_dirs=[numpy_include_dir]
添加到这些扩展中。
Edit : As discussed in the chat: The error was due to the conda env ignoring the CC variable.编辑:正如聊天中所讨论的:错误是由于 conda env 忽略了 CC 变量。 After installing python+pip via homebrew and the required python packages via pip, this answer's steps worked for the OP.通过自制软件安装 python+pip 并通过 pip 安装所需的 python 包后,此答案的步骤适用于 OP。
All in all, here's a setup.py that worked for me (macOS 10.5.7, gcc-10):总而言之,这是一个对我有用的 setup.py(macOS 10.5.7,gcc-10):
try:
from setuptools import setup
except ImportError:
from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension
import numpy
# Get the numpy include directory.
numpy_include_dir = numpy.get_include()
# Extensions
# pykdtree (kd tree)
pykdtree = Extension(
'im2mesh.utils.libkdtree.pykdtree.kdtree',
sources=[
'im2mesh/utils/libkdtree/pykdtree/kdtree.c',
'im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c'
],
language='c',
extra_compile_args=['-std=c99', '-O3', '-fopenmp'],
extra_link_args=['-lgomp'],
include_dirs=[numpy_include_dir]
)
# mcubes (marching cubes algorithm)
mcubes_module = Extension(
'im2mesh.utils.libmcubes.mcubes',
sources=[
'im2mesh/utils/libmcubes/mcubes.pyx',
'im2mesh/utils/libmcubes/pywrapper.cpp',
'im2mesh/utils/libmcubes/marchingcubes.cpp'
],
language='c++',
extra_compile_args=['-std=c++11'],
include_dirs=[numpy_include_dir]
)
# triangle hash (efficient mesh intersection)
triangle_hash_module = Extension(
'im2mesh.utils.libmesh.triangle_hash',
sources=[
'im2mesh/utils/libmesh/triangle_hash.pyx'
],
libraries=['m'], # Unix-like specific
include_dirs=[numpy_include_dir]
)
# mise (efficient mesh extraction)
mise_module = Extension(
'im2mesh.utils.libmise.mise',
sources=[
'im2mesh/utils/libmise/mise.pyx'
],
)
# simplify (efficient mesh simplification)
simplify_mesh_module = Extension(
'im2mesh.utils.libsimplify.simplify_mesh',
sources=[
'im2mesh/utils/libsimplify/simplify_mesh.pyx'
],
include_dirs=[numpy_include_dir]
)
# voxelization (efficient mesh voxelization)
voxelize_module = Extension(
'im2mesh.utils.libvoxelize.voxelize',
sources=[
'im2mesh/utils/libvoxelize/voxelize.pyx'
],
libraries=['m'] # Unix-like specific
)
# DMC extensions
dmc_pred2mesh_module = CppExtension(
'im2mesh.dmc.ops.cpp_modules.pred2mesh',
sources=[
'im2mesh/dmc/ops/cpp_modules/pred_to_mesh_.cpp',
]
)
# dmc_cuda_module = CUDAExtension(
# 'im2mesh.dmc.ops._cuda_ext',
# sources=[
# 'im2mesh/dmc/ops/src/extension.cpp',
# 'im2mesh/dmc/ops/src/curvature_constraint_kernel.cu',
# 'im2mesh/dmc/ops/src/grid_pooling_kernel.cu',
# 'im2mesh/dmc/ops/src/occupancy_to_topology_kernel.cu',
# 'im2mesh/dmc/ops/src/occupancy_connectivity_kernel.cu',
# 'im2mesh/dmc/ops/src/point_triangle_distance_kernel.cu',
# ]
# )
# Gather all extension modules
ext_modules = [
pykdtree,
mcubes_module,
triangle_hash_module,
mise_module,
simplify_mesh_module,
voxelize_module,
dmc_pred2mesh_module,
#dmc_cuda_module,
]
setup(
ext_modules=cythonize(ext_modules),
cmdclass={
'build_ext': BuildExtension
}
)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.