如何在 Amazon EMR 上安裝多個版本的 numpy 以及如何刪除早期版本？

Question

我不明白 Python 如何可以安裝單個 package 的多個版本，或者為什么，當我安裝了多個版本時， import package沒有給我最新的一個。

我在 AWS EMR 中使用 AWS linux 和 AWS 存儲庫。 當我安裝 Python 3.6 時，它默認為我提供 Numpy 版本 1.14.5。 我想使用版本 1.19.1 並將它放在一個目錄中。

$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/packages/ numpy==1.19.1
Requirement already up-to-date: numpy==1.19.1 in /usr/local/lib64/python3.6/site-packages
$

但請注意：

 $ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.14.5'
>>>

因為我同時安裝了 1.14.5 和 1.19.1 版本。

我可以卸載 1.19.1 版，然后我可以卸載 1.14.5 版，雖然我似乎必須卸載它兩次（請參閱下面的擴展混音）。

那么如何讓import numpy獲得版本1.19.1而不是舊版本？

$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib64/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib64/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib64/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib64/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/multiarray_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/operand_flag_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/struct_ufunc_test.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/test_rational.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/core/umath_tests.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/__version__.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/compat.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/environment.py
  /usr/local/lib64/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/f2py/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/fftpack_lite.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib64/python3.6/site-packages/numpy/fft/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/fft/tests/test_fftpack.py
  /usr/local/lib64/python3.6/site-packages/numpy/lib/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/linalg/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/ma/version.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/info.py
  /usr/local/lib64/python3.6/site-packages/numpy/random/randomkit.h
  /usr/local/lib64/python3.6/site-packages/numpy/testing/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/__init__.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/decorators.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/nosetester.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/parameterized.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nose_tools/utils.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/noseclasses.py
  /usr/local/lib64/python3.6/site-packages/numpy/testing/nosetester.py
Proceed (y/n)? y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>>
$ sudo pip-3.6 uninstall numpy
Uninstalling numpy-1.14.5:
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.data/scripts/f2py
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/METADATA
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/RECORD
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/WHEEL
  /usr/local/lib/python3.6/site-packages/numpy-1.14.5.dist-info/top_level.txt
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libgfortran-ed201abd.so.3.0.0
  /usr/local/lib/python3.6/site-packages/numpy/.libs/libopenblasp-r0-39a31c03.2.18.so
  /usr/local/lib/python3.6/site-packages/numpy/LICENSE.txt
  /usr/local/lib/python3.6/site-packages/numpy/__config__.py
  /usr/local/lib/python3.6/site-packages/numpy/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/_distributor_init.py
  /usr/local/lib/python3.6/site-packages/numpy/_globals.py
  /usr/local/lib/python3.6/site-packages/numpy/_import_tools.py
  /usr/local/lib/python3.6/site-packages/numpy/add_newdocs.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/_inspect.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/py3k.py
  /usr/local/lib/python3.6/site-packages/numpy/compat/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/conftest.py
  /usr/local/lib/python3.6/site-packages/numpy/core/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_dummy.cpython-36m-x86_64-linux-gnu.so
  /usr/local/lib/python3.6/site-packages/numpy/core/_internal.py
  /usr/local/lib/python3.6/site-packages/numpy/core/_methods.py
  /usr/local/lib/python3.6/site-packages/numpy/core/arrayprint.py
  /usr/local/lib/python3.6/site-packages/numpy/core/cversions.py
  /usr/local/lib/python3.6/site-packages/numpy/core/defchararray.py
  /usr/local/lib/python3.6/site-packages/numpy/core/einsumfunc.py
  /usr/local/lib/python3.6/site-packages/numpy/core/fromnumeric.py
  /usr/local/lib/python3.6/site-packages/numpy/core/function_base.py
  /usr/local/lib/python3.6/site-packages/numpy/core/generate_numpy_api.py
  /usr/local/lib/python3.6/site-packages/numpy/core/getlimits.py
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__multiarray_api.h
  /usr/local/lib/python3.6/site-packages/numpy/core/include/numpy/__ufunc_api.h
...
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/lahey.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/mips.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/nag.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/none.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pathf95.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/pg.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/sun.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/fcompiler/vast.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/from_template.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/intelccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/lib2def.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/line_endings.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/log.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw/gfortran_vs2003_hack.c
  /usr/local/lib/python3.6/site-packages/numpy/distutils/mingw32ccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvc9compiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/msvccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/numpy_distribution.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/pathccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_exec_command.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_gnu.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_intel.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_fcompiler_nagfor.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_misc_util.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_npy_pkg_config.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/tests/test_system_info.py
  /usr/local/lib/python3.6/site-packages/numpy/distutils/unixccompiler.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/basics.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/broadcasting.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/byteswapping.py
  /usr/local/lib/python3.6/site-packages/numpy/doc/constants.py
...
  /usr/local/lib/python3.6/site-packages/numpy/testing/print_coercion_tables.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/setup.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_decorators.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_doctesting.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/tests/test_utils.py
  /usr/local/lib/python3.6/site-packages/numpy/testing/utils.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/__init__.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_ctypeslib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_matlib.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_numpy_version.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_reloading.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_scripts.py
  /usr/local/lib/python3.6/site-packages/numpy/tests/test_warnings.py
  /usr/local/lib/python3.6/site-packages/numpy/version.py
Proceed (y/n)? Y
  Successfully uninstalled numpy-1.14.5
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ sudo pip-3.6 uninstall numpy
Cannot uninstall requirement numpy, not installed
You are using pip version 9.0.3, however version 20.2.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'numpy'
>>>
$ sudo pip-3.6 install --upgrade --no-index --find-links /mnt/gits/das-vm-config/dist/python numpy==1.19.1
Collecting numpy==1.19.1
Installing collected packages: numpy
Successfully installed numpy-1.19.1
$ python
Python 3.6.11 (default, Jul 20 2020, 22:15:17)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy
>>> numpy.version.version
'1.19.1'
>>>

Answer 1

我們的解決方案：

1. 使用yum安裝python36和python36-pip
2. 使用帶有--target選項的 pip-3.6 在/mnt/python_additions中安裝numpy-1.19.1 。 我們在 Bootstrap 操作中執行此操作，以便它發生在每個節點上。
3. 讓 Amazon 的腳本安裝python36-numpy ，它將numpy-1.14.4安裝在站點庫位置。
4. 在從spark-submit調用的程序中，將/mnt/python_additions添加到sys.path 。

這是一個黑客，但它的工作原理。

Answer 2

我剛剛遇到了同樣的問題。 我最終運行了以下 3 次以卸載所有 numpy 版本：

python3 -m pip uninstall numpy

一旦它告訴我沒有安裝 numpy，我運行安裝，它能夠給我正確的版本。

很煩人，以前不是問題。

Answer 3

雖然您可以安裝多個版本的 package，但在 Python 環境中只能導入一個版本。 如果您想明確說明您要使用的 package 版本，請查看 Python 從何處導入包，您可以在交互式 Z21D6F40CFB511982E45724E0E2509 中運行以下命令：

>>> import sys
>>> print(sys.path)

這將返回一個導入位置列表。 列表中的最后一項顯示 Python 將查找的最后一個位置。 列表中后面的路徑會取代前面的路徑。 如果您有以下情況：

["site_packages_path_with_numpy_1.19.1", "site_packages_with_numpy_1.14.5"]

Python 將導入 numpy 1.14.5。

如果您還沒有遇到它們，我建議您查看virtualenv和requirements.txt 。

Virtualenv 允許您使用所需的任何包創建 Python 的本地版本（與系統版本分開）。 例如。

python -m venv .venv
source .venv/bin/activate

這將創建 Python 的新副本並激活它。 對 Python 或 PIP 的調用現在將使用虛擬環境。

您可以像這樣將軟件包安裝到虛擬環境中。

pip install numpy==1.19.1

要求允許您保留要安裝的軟件包的列表。 文件看起來像這樣

# requirements.txt

numpy==1.19.1

您可以像這樣將需求的內容安裝到虛擬環境中（假設您已經運行了活動命令）：

pip install -r requirements.txt

最后，一個更現代且可以說更好的解決方案是https://python-poetry.org/ 。