安装 PySpark 后 Python 版本中的问题
[英]Issue in Python version after installing PySpark
问题描述
I have installed PySpark 3.1.2 along with OpenJDK-1.8 to connect with a docker instance of Cassandra 4.0.1.
我已经安装了 PySpark 3.1.2 和 OpenJDK-1.8 以连接 Cassandra 4.0.1 的 docker 实例。 I followed the instructions as in https://towardsdatascience.com/installing-pyspark-with-java-8-on-ubuntu-18-04-6a9dea915b5b and successfully installed the required versions.我按照https://towardsdatascience.com/installing-pyspark-with-java-8-on-ubuntu-18-04-6a9dea915b5b 中的说明操作并成功安装了所需的版本。
I'm using anaconda environment, after installation I noticed that my Python version got automatically downgraded to 3.5 which is not supported by Pyspark(even in all environments where I had different python versions earlier, it's now 3.5).我正在使用 anaconda 环境，安装后我注意到我的 Python 版本自动降级到 Pyspark 不支持的 3.5（即使在我之前有不同 Python 版本的所有环境中，现在都是 3.5）。 I read that Pyspark needs python3.6+.我读到 Pyspark 需要 python3.6+。 I tried everything possible to upgrade the python version to 3.6+ but it's not happening.我尝试了一切可能将 python 版本升级到 3.6+，但它没有发生。 When I try conda upgrade python some upgrades and removals happen but python is still 3.5.当我尝试conda upgrade python 时，会发生一些升级和删除，但 python 仍然是 3.5。
conda update python gives: conda 更新 python给出：
Collecting package metadata (current_repodata.json): done
Solving environment: \
Updating python is constricted by
anaconda -> requires python==3.8.3=hcff3b4d_2
If you are sure you want an update of your package either try `conda update --all` or install a specific version of the package you want using `conda install <pkg>=<version>`
done
# All requested packages already installed.
While using conda install python=3.7 after installing/upgrading so many packages the following error message is obtained:在安装/升级这么多包后使用 conda install python=3.7 时，获得以下错误消息：
Downloading and Extracting Packages
traitlets-4.3.3 | 140 KB | ##################################### | 100%
conda-token-0.3.0 | 10 KB | ##################################### | 100%
testpath-0.4.4 | 82 KB | ##################################### | 100%
click-7.1.2 | 71 KB | ##################################### | 100%
singledispatch-3.4.0 | 16 KB | ##################################### | 100%
conda-package-handli | 881 KB | ##################################### | 100%
astropy-4.0.1.post1 | 6.1 MB | ##################################### | 100%
greenlet-0.4.16 | 24 KB | ##################################### | 100%
navigator-updater-0. | 691 KB | ##################################### | 100%
defusedxml-0.6.0 | 23 KB | ##################################### | 100%
importlib-metadata-1 | 51 KB | ##################################### | 100%
ipython-7.16.1 | 987 KB | ##################################### | 100%
numba-0.50.1 | 3.1 MB | ##################################### | 100%
mkl_fft-1.1.0 | 143 KB | ##################################### | 100%
pycurl-7.43.0.5 | 68 KB | ##################################### | 100%
scikit-learn-0.23.1 | 5.0 MB | ##################################### | 100%
sympy-1.6.1 | 8.6 MB | ##################################### | 100%
matplotlib-base-3.2. | 5.4 MB | ##################################### | 100%
pandocfilters-1.4.2 | 13 KB | ##################################### | 100%
pkginfo-1.5.0.1 | 44 KB | ##################################### | 100%
ipykernel-5.3.2 | 179 KB | ##################################### | 100%
anaconda-navigator-2 | 5.1 MB | ##################################### | 100%
pylint-2.5.3 | 441 KB | ##################################### | 100%
kiwisolver-1.2.0 | 84 KB | ##################################### | 100%
setuptools-49.2.0 | 743 KB | ##################################### | 100%
bokeh-2.1.1 | 5.4 MB | ##################################### | 100%
cryptography-2.9.2 | 552 KB | ##################################### | 100%
zope.interface-4.7.1 | 204 KB | ##################################### | 100%
ujson-1.35 | 25 KB | ##################################### | 100%
_ipyw_jlab_nb_ext_co | 4 KB | ##################################### | 100%
asn1crypto-1.3.0 | 164 KB | ##################################### | 100%
pyzmq-19.0.1 | 459 KB | ##################################### | 100%
terminado-0.8.3 | 26 KB | ##################################### | 100%
conda-content-trust- | 56 KB | ##################################### | 100%
jsonschema-3.2.0 | 91 KB | ##################################### | 100%
bitarray-1.4.0 | 88 KB | ##################################### | 100%
beautifulsoup4-4.9.1 | 165 KB | ##################################### | 100%
certifi-2020.6.20 | 156 KB | ##################################### | 100%
nbconvert-5.6.1 | 451 KB | ##################################### | 100%
zope.event-4.4 | 10 KB | ##################################### | 100%
cffi-1.14.0 | 224 KB | ##################################### | 100%
path-13.1.0 | 35 KB | ##################################### | 100%
scikit-image-0.16.2 | 23.1 MB | ##################################### | 100%
python-language-serv | 81 KB | ##################################### | 100%
werkzeug-1.0.1 | 240 KB | ##################################### | 100%
conda-repo-cli-1.0.4 | 47 KB | ##################################### | 100%
anaconda-2020.07 | 17 KB | ##################################### | 100%
numpy-1.18.5 | 5 KB | ##################################### | 100%
flask-1.1.2 | 78 KB | ##################################### | 100%
psutil-5.7.0 | 319 KB | ##################################### | 100%
jinja2-2.11.2 | 103 KB | ##################################### | 100%
gevent-20.6.2 | 1.6 MB | ##################################### | 100%
six-1.15.0 | 13 KB | ##################################### | 100%
distributed-2.20.0 | 992 KB | ##################################### | 100%
wheel-0.34.2 | 51 KB | ##################################### | 100%
numpy-base-1.18.5 | 4.1 MB | ##################################### | 100%
lxml-4.5.2 | 1.2 MB | ##################################### | 100%
pip-20.1.1 | 1.7 MB | ##################################### | 100%
isort-4.3.21 | 69 KB | ##################################### | 100%
spyder-kernels-1.9.2 | 97 KB | ##################################### | 100%
pandas-1.0.5 | 7.8 MB | ##################################### | 100%
spyder-4.1.4 | 5.6 MB | ##################################### | 100%
python-3.7.7 | 45.1 MB | ##################################### | 100%
pytest-5.4.3 | 388 KB | ##################################### | 100%
backports-1.0 | 160 KB | ##################################### | 100%
conda-build-3.21.5 | 549 KB | ##################################### | 100%
sqlalchemy-1.3.18 | 1.5 MB | ##################################### | 100%
cytoolz-0.10.1 | 376 KB | ##################################### | 100%
pillow-7.2.0 | 617 KB | ##################################### | 100%
notebook-6.0.3 | 4.0 MB | ##################################### | 100%
keyring-21.2.1 | 58 KB | ##################################### | 100%
conda-4.10.3 | 2.9 MB | ##################################### | 100%
statsmodels-0.11.1 | 7.8 MB | ##################################### | 100%
decorator-4.4.2 | 14 KB | ##################################### | 100%
scipy-1.5.0 | 14.4 MB | ##################################### | 100%
llvmlite-0.33.0 | 17.3 MB | ##################################### | 100%
pywavelets-1.1.1 | 3.5 MB | ##################################### | 100%
regex-2020.6.8 | 324 KB | ##################################### | 100%
pyrsistent-0.16.0 | 94 KB | ##################################### | 100%
Preparing transaction: failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
exit_code = do_call(args, p)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 83, in do_call
return getattr(module, func_name)(args, parser)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
install(args, parser, 'install')
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 317, in install
handle_txn(unlink_link_transaction, prefix, args, newenv)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 346, in handle_txn
unlink_link_transaction.execute()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 245, in execute
self.verify()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 222, in verify
self.prepare()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 211, in prepare
grps = self._prepare(self.transaction_context, stp.target_prefix,
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in _prepare
packages_info_to_link = tuple(read_package_info(prec, pcrec)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in <genexpr>
packages_info_to_link = tuple(read_package_info(prec, pcrec)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 89, in read_package_info
package_metadata = read_package_metadata(epd)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 144, in read_package_metadata
data = json.loads(f.read())
File "/home/xx/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
`$ /home/xx/anaconda3/bin/conda install python=3.7`
environment variables:
CIO_TEST=<not set>
COMPIZ_BIN_PATH=/usr/bin/
CONDA_DEFAULT_ENV=base
CONDA_EXE=/home/xx/anaconda3/bin/conda
CONDA_PREFIX=/home/xx/anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/home/xx/anaconda3/bin/python
CONDA_ROOT=/home/xx/anaconda3
CONDA_SHLVL=1
CURL_CA_BUNDLE=<not set>
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
PATH=/home/xx/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/u
sr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/xx/s
park-3.1.2-bin-
hadoop3.2/bin:/home/xx/anaconda3/bin:/usr/lib/jvm/java-8-openjdk-
amd64/jre/bin
PYTHONPATH=/home/xx/spark-3.1.2-bin-hadoop3.2/python:
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
active environment : base
active env location : /home/xx/anaconda3
shell level : 1
user config file : /home/xx/.condarc
populated config files :
conda version : 4.10.3
conda-build version : 3.18.11
python version : 3.8.3.final.0
virtual packages : __linux=4.15.0=0
__glibc=2.23=0
__unix=0=0
__archspec=1=x86_64
base environment : /home/xx/anaconda3 (writable)
conda av data dir : /home/xx/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/xx/anaconda3/pkgs
/home/xx/.conda/pkgs
envs directories : /home/xx/anaconda3/envs
/home/xx/.conda/envs
platform : linux-64
user-agent : conda/4.10.3 requests/2.24.0 CPython/3.8.3 Linux/4.15.0-142-generic ubuntu/16.04.7 glibc/2.23
UID:GID : 1000:1000
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Can someone help me figure out the issue?有人可以帮我弄清楚这个问题吗？ How can I resolve this and upgrade the Python package?如何解决此问题并升级 Python 包？
1 个解决方案
解决方案1
0 已采纳 2021-11-08 16:26:56
I resolved the issue by manually installing pyspark and making a minor change in the environment variables.我通过手动安装 pyspark 并在环境变量中进行微小更改来解决该问题。
After downloading the required version of spark, you need to configure environment variables.下载所需版本的spark后，需要配置环境变量。 There are a few Spark home paths you need to add to the user profile as follows,您需要将一些 Spark 主路径添加到用户配置文件中，如下所示，
echo "export SPARK_HOME=/opt/spark" >> ~/.profile
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.profile
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile
Here, for me just after adding the spark home path and other parameters my python version downgrades to 3.5 in anaconda.在这里，对我来说，在添加 spark 主路径和其他参数后，我的 python 版本在 anaconda 中降级到 3.5。
So I changed the Python path in the user profile to: PYSPARK_PYTHON=/usr/bin/python3.7于是我将用户配置文件中的Python路径改为： PYSPARK_PYTHON=/usr/bin/python3.7
which resolved the issue since pyspark is compatible with python3.6+解决了这个问题，因为 pyspark 与 python3.6+ 兼容
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
Python-在python 2.6.6版本中安装Argpath的问题 - Python - Issue in Installing Argpath for python 2.6.6 version
安装django后更改python版本 - Change the python version after installing django
使用port安装numpy（Python默认版本问题） - Installing numpy using port (Python default version issue)
在python2.7.5和旧版本问题上安装pip - installing pip on python2.7.5 and the old version issue
在OSX上pip安装MySQL-python之后的版本错误 - Version error after pip installing MySQL-python on OSX
即使用当前的python版本安装它后，Numpy导入错误 - Numpy import error even after installing it with current python version
安装pyenv-win后本地python版本没有改变 - Local python version not changing after installing pyenv-win
安装不同版本并更改路径变量后出现“No Python at ...”错误 - "No Python at..." error after installing different version and changing the path variable
安装本地版本的Python - Installing Local Version of Python
安装旧版本的python - Installing an older version of python
相关标签