pipenv locking for NumPy fails on Python3.9 and pip21.0.1

Question

When I try to install pandas via pipenv I get a bunch of numpy errors after the locking process. When I try to install bs4 via pipenv it works fine, but for some reason, it fails with pandas. I have tried deleting and reinstalling pipenv and removing the environment, but nothing has worked.

I am using Python 3.9.1 and pip 21.0.1.
I am on macOS Big Sur.

Here is the code snippet of the errors I get. The errors occur during the locking process (writing to pipenv.lock):

  Copying numpy.egg-info to build/bdist.macosx-11-x86_64/wheel/numpy-1.20.0-py3.9.egg-info
  running install_scripts
  Traceback (most recent call last):
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 257, in <module>
      main()
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 240, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_vendor/pep517/_in_process.py", line 181, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 211, in build_wheel
      return self._build_with_temp_dir(['bdist_wheel'], '.whl',
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 197, in _build_with_temp_dir
      self.run_setup()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 248, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/build_meta.py", line 142, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 513, in <module>
      setup_package()
    File "setup.py", line 505, in setup_package
      setup(**metadata)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/numpy/distutils/core.py", line 169, in setup
      return old_setup(**new_attr)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/setuptools/__init__.py", line 165, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/core.py", line 148, in setup
      dist.run_commands()
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 966, in run_commands
      self.run_command(cmd)
    File "/usr/local/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/distutils/dist.py", line 985, in run_command
      cmd_obj.run()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 328, in run
      impl_tag, abi_tag, plat_tag = self.get_tag()
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 252, in get_tag
      plat_name = get_platform(self.bdist_dir)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 48, in get_platform
      result = calculate_macosx_platform_tag(archive_root, result)
    File "/private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-9iqn62cr/overlay/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 356, in calculate_macosx_platform_tag
      assert len(base_version) == 2
  AssertionError

  ########### EXT COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : clang

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   :
                :
    SSE41       : SSE SSE2 SSE3 SSSE3
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1
    Extra checks: none
    Detect      : SSE SSE2 SSE3 SSSE3 SSE41
                : numpy/core/src/umath/_umath_tests.dispatch.c
                :
    SSE42       : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2
    Extra checks: none
    Detect      : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX2        : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mavx2
    Extra checks: none
    Detect      : AVX F16C AVX2
                : numpy/core/src/umath/_umath_tests.dispatch.c
                :
    (FMA3 AVX2) : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2
    Extra checks: none
    Detect      : AVX F16C FMA3 AVX2
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX512F     : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f
    Extra checks: AVX512F_REDUCE
    Detect      : AVX512F
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
                :
    AVX512_SKX  : SSE SSE2 SSE3 SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD
    Flags       : -msse -msse2 -msse3 -mssse3 -msse4.1 -mpopcnt -msse4.2 -mavx -mf16c -mfma -mavx2 -mavx512f -mavx512cd -mavx512vl -mavx512bw -mavx512dq
    Extra checks: AVX512BW_MASK AVX512DQ_MASK
    Detect      : AVX512_SKX
                : build/src.macosx-11-x86_64-3.9/numpy/core/src/_simd/_simd.dispatch.c
  CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_ext.py

  ########### CLIB COMPILER OPTIMIZATION ###########
  Platform      :
    Architecture: x64
    Compiler    : clang

  CPU baseline  :
    Requested   : 'min'
    Enabled     : SSE SSE2 SSE3
    Flags       : -msse -msse2 -msse3
    Extra checks: none

  CPU dispatch  :
    Requested   : 'max -xop -fma4'
    Enabled     : SSSE3 SSE41 POPCNT SSE42 AVX F16C FMA3 AVX2 AVX512F AVX512CD AVX512_KNL AVX512_SKX AVX512_CLX AVX512_CNL AVX512_ICL
    Generated   : none
  CCompilerOpt._cache_write[796] : write cache to path -> /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-install-o5k4mrv9/numpy/build/temp.macosx-11-x86_64-3.9/ccompiler_opt_cache_clib.py
  ----------------------------------------
  ERROR: Failed building wheel for numpy
Failed to build numpy
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
----------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 764, in <module>
    main()
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 758, in main
    _main(parsed.pre, parsed.clear, parsed.verbose, parsed.system, parsed.write,
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 741, in _main
    resolve_packages(pre, clear, verbose, system, write, requirements_dir, packages, dev)
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 702, in resolve_packages
    results, resolver = resolve(
  File "/usr/local/lib/python3.9/site-packages/pipenv/resolver.py", line 684, in resolve
    return resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1395, in resolve_deps
    results, hashes, markers_lookup, resolver, skipped = actually_resolve_deps(
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 1108, in actually_resolve_deps
    resolver.resolve()
  File "/usr/local/lib/python3.9/site-packages/pipenv/utils.py", line 823, in resolve
    results = self.resolver.resolve(max_rounds=environments.PIPENV_MAX_ROUNDS)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 180, in resolve
    has_changed, best_matches = self._resolve_one_round()
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 268, in _resolve_one_round
    their_constraints.extend(self._iter_dependencies(best_match))
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/resolver.py", line 383, in _iter_dependencies
    dependencies = self.repository.get_dependencies(ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 229, in get_dependencies
    legacy_results = self.get_legacy_dependencies(ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 350, in get_legacy_dependencies
    results, ireq = self.resolve_reqs(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/piptools/repositories/pypi.py", line 306, in resolve_reqs
    results = resolver._resolve_one(reqset, ireq)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 339, in _resolve_one
    abstract_dist = self._get_abstract_dist_for(req_to_install)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/legacy_resolve.py", line 287, in _get_abstract_dist_for
    abstract_dist = self.preparer.prepare_linked_requirement(req)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 508, in prepare_linked_requirement
    abstract_dist = _get_prepared_distribution(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/operations/prepare.py", line 95, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(finder, build_isolation)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 38, in prepare_distribution_metadata
    self._setup_isolation(finder)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/distributions/sdist.py", line 66, in _setup_isolation
    self.req.build_env.install_requirements(
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/build_env.py", line 202, in install_requirements
    call_subprocess(args, spinner=spinner)
  File "/usr/local/lib/python3.9/site-packages/pipenv/patched/notpip/_internal/utils/subprocess.py", line 242, in call_subprocess
    raise InstallationError(exc_msg)
pipenv.patched.notpip._internal.exceptions.InstallationError: Command errored out with exit status 1: /Users/kaiznanji/.local/share/virtualenvs/Kalgo-wH0XXFPH/bin/python /usr/local/lib/python3.9/site-packages/pipenv/patched/notpip install --ignore-installed --no-user --prefix /private/var/folders/d9/737381n90cv4_jc2pnx1vc5w0000gn/T/pip-build-env-4nfnxp9e/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- setuptools wheel 'Cython>=0.29.21,<3' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system!='"'"'AIX'"'"'' 'numpy==1.16.5; python_version=='"'"'3.7'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy==1.17.3; python_version=='"'"'3.8'"'"' and platform_system=='"'"'AIX'"'"'' 'numpy; python_version>='"'"'3.9'"'"'' Check the logs for full command output.

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
bs4 = "*"
requests = "*"
pandas = "*"
joblib = "*"

[dev-packages]

[requires]
python_version = "3.9"

Answer 1

The stacktrace from the original poster as well as mine included an error in bdist_wheel.py. Opening that exact file, it was importing 'distutils' to call 'distutils.util.get_platform()'. But when I started a python shell to see what that call returned on my machine, the active version of distutils didn't have 'util' subpackage. That's a clue that the python version I'm using is not the version being used by pipenv to build libraries.

That led me to figure out what version of python I really wanted to use and make sure pipenv was using that as well as currently installed packages:

> which python
/usr/local/Caskroom/miniforge/base/bin/python

>pipenv install --python=/usr/local/Caskroom/miniforge/base/bin/python  --site-packages

This works so far and may help some folks, however specifying the python version does not work for pipenv lock so it's probably not a complete solution.

Ultimately I probably need to go back to basics and use pyenv to properly manage the python version and keep things consistent.

Answer 2

I face this problem constantly. Sometimes this solution works, but not all the time. Run this in your terminal:

 export SYSTEM_VERSION_COMPAT=1

If someone else understands why this works sometimes, please feel free to comment or edit my answer.

Answer 3

Running this first solved it for me: export SYSTEM_VERSION_COMPAT=1