Cannot install lightgbm==3.3.3 on Apple Silicon

Question

Here the full log of pip3 install lightgbm==3.3.3 .

me % pip3 install lightgbm==3.3.3
Collecting lightgbm==3.3.3
  Using cached lightgbm-3.3.3.tar.gz (1.5 MB)
  Preparing metadata (setup.py) ... done
Requirement already satisfied: wheel in /opt/homebrew/lib/python3.10/site-packages (from lightgbm==3.3.3) (0.37.1)
Collecting numpy
  Using cached numpy-1.23.5-cp310-cp310-macosx_11_0_arm64.whl (13.4 MB)
Collecting scipy
  Using cached scipy-1.9.3-cp310-cp310-macosx_12_0_arm64.whl (28.5 MB)
Collecting scikit-learn!=0.22.0
  Downloading scikit_learn-1.1.3-cp310-cp310-macosx_12_0_arm64.whl (7.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 7.7/7.7 MB 4.7 MB/s eta 0:00:00
Collecting threadpoolctl>=2.0.0
  Downloading threadpoolctl-3.1.0-py3-none-any.whl (14 kB)
Collecting joblib>=1.0.0
  Using cached joblib-1.2.0-py3-none-any.whl (297 kB)
Building wheels for collected packages: lightgbm
  Building wheel for lightgbm (setup.py) ... error
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [86 lines of output]
      running bdist_wheel
      /opt/homebrew/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      running build
      running build_py
      creating build
      creating build/lib
      creating build/lib/lightgbm
      copying lightgbm/callback.py -> build/lib/lightgbm
      copying lightgbm/compat.py -> build/lib/lightgbm
      copying lightgbm/plotting.py -> build/lib/lightgbm
      copying lightgbm/__init__.py -> build/lib/lightgbm
      copying lightgbm/engine.py -> build/lib/lightgbm
      copying lightgbm/dask.py -> build/lib/lightgbm
      copying lightgbm/basic.py -> build/lib/lightgbm
      copying lightgbm/libpath.py -> build/lib/lightgbm
      copying lightgbm/sklearn.py -> build/lib/lightgbm
      running egg_info
      writing lightgbm.egg-info/PKG-INFO
      writing dependency_links to lightgbm.egg-info/dependency_links.txt
      writing requirements to lightgbm.egg-info/requires.txt
      writing top-level names to lightgbm.egg-info/top_level.txt
      reading manifest file 'lightgbm.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      no previously-included directories found matching 'build'
      warning: no files found matching '*.so' under directory 'lightgbm'
      warning: no files found matching '*.so' under directory 'compile'
      warning: no files found matching '*.dll' under directory 'compile/Release'
      warning: no files found matching '*.dll' under directory 'compile/windows/x64/DLL'
      warning: no previously-included files matching '*.py[co]' found anywhere in distribution
      warning: no previously-included files found matching 'compile/external_libs/compute/.git'
      adding license file 'LICENSE'
      writing manifest file 'lightgbm.egg-info/SOURCES.txt'
      copying lightgbm/VERSION.txt -> build/lib/lightgbm
      installing to build/bdist.macosx-13-arm64/wheel
      running install
      INFO:LightGBM:Starting to compile the library.
      INFO:LightGBM:Starting to compile with CMake.
      Traceback (most recent call last):
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 95, in silent_call
          subprocess.check_call(cmd, stderr=log, stdout=log)
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 364, in check_call
          retcode = call(*popenargs, **kwargs)
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 345, in call
          with Popen(*popenargs, **kwargs) as p:
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'cmake'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 334, in <module>
          setup(name='lightgbm',
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/opt/homebrew/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 335, in run
          self.run_command('install')
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
          self.distribution.run_command(command)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 248, in run
          compile_cpp(use_mingw=self.mingw, use_gpu=self.gpu, use_cuda=self.cuda, use_mpi=self.mpi,
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 198, in compile_cpp
          silent_call(cmake_cmd, raise_error=True, error_msg='Please install CMake and all required dependencies first')
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 99, in silent_call
          raise Exception("\n".join((error_msg, LOG_NOTICE)))
      Exception: Please install CMake and all required dependencies first
      The full version of error log was saved into /Users/me/LightGBM_compilation.log
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for lightgbm
  Running setup.py clean for lightgbm
Failed to build lightgbm
Installing collected packages: threadpoolctl, numpy, joblib, scipy, scikit-learn, lightgbm
  Running setup.py install for lightgbm ... error
  error: subprocess-exited-with-error
  
  × Running setup.py install for lightgbm did not run successfully.
  │ exit code: 1
  ╰─> [45 lines of output]
      running install
      /opt/homebrew/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
        warnings.warn(
      INFO:LightGBM:Starting to compile the library.
      INFO:LightGBM:Starting to compile with CMake.
      Traceback (most recent call last):
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 95, in silent_call
          subprocess.check_call(cmd, stderr=log, stdout=log)
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 364, in check_call
          retcode = call(*popenargs, **kwargs)
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 345, in call
          with Popen(*popenargs, **kwargs) as p:
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 971, in __init__
          self._execute_child(args, executable, preexec_fn, close_fds,
        File "/opt/homebrew/Cellar/python@3.10/3.10.8/Frameworks/Python.framework/Versions/3.10/lib/python3.10/subprocess.py", line 1847, in _execute_child
          raise child_exception_type(errno_num, err_msg, err_filename)
      FileNotFoundError: [Errno 2] No such file or directory: 'cmake'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 334, in <module>
          setup(name='lightgbm',
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/__init__.py", line 87, in setup
          return distutils.core.setup(**attrs)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 185, in setup
          return run_commands(dist)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
          dist.run_commands()
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
          self.run_command(cmd)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/dist.py", line 1217, in run_command
          super().run_command(command)
        File "/opt/homebrew/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
          cmd_obj.run()
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 248, in run
          compile_cpp(use_mingw=self.mingw, use_gpu=self.gpu, use_cuda=self.cuda, use_mpi=self.mpi,
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 198, in compile_cpp
          silent_call(cmake_cmd, raise_error=True, error_msg='Please install CMake and all required dependencies first')
        File "/private/var/folders/qf/7kpp7kws2bs9ljbxdl6vf9480000gn/T/pip-install-qdmnvrdo/lightgbm_7e71affc27c54e8fb3f78d9ef73bd942/setup.py", line 99, in silent_call
          raise Exception("\n".join((error_msg, LOG_NOTICE)))
      Exception: Please install CMake and all required dependencies first
      The full version of error log was saved into /Users/me/LightGBM_compilation.log
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> lightgbm

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.

There is a workaround for this?

Answer 1

When you run pip install lightgbm and see this message in logs:

Building wheels for collected packages: lightgbm

it means that there is not a pre-compiled binary (ie wheel) available matching your platform (operating system + architecture + Python version), and that LightGBM needs to be built from source.

lightgbm is a Python package wrapping lib_lightgbm , a C++ library with a C API. So "built from source" for lightgbm means compiling that C/C++ code, which for LightGBM requires:

• C and C++ compilers
• the CMake build system
• an installation of OpenMP

Those components are the "CMake and all required dependencies" referred to in the error message.

On macOS, you should already have a C/C++ compiler installed ( clang ) by default. To get CMake and OpenMP, run the following.

brew install cmake libomp

NOTE: lightgbm v3.3.3 and older does not support the newest version of OpenMP available as of this writing (v15.x). That was fixed in microsoft/LightGBM#5563 . If you end up with OpenMP >=15.0 and lightgbm>=4.0 is not yet available from PyPI, either downgrade OpenMP or build a development version of lightgbm (see "Install from GitHub" in the docs).