Issue in Python version after installing PySpark
Question
I have installed PySpark 3.1.2 along with OpenJDK-1.8 to connect with a docker instance of Cassandra 4.0.1. I followed the instructions as in https://towardsdatascience.com/installing-pyspark-with-java-8-on-ubuntu-18-04-6a9dea915b5b and successfully installed the required versions.
I'm using anaconda environment, after installation I noticed that my Python version got automatically downgraded to 3.5 which is not supported by Pyspark(even in all environments where I had different python versions earlier, it's now 3.5). I read that Pyspark needs python3.6+. I tried everything possible to upgrade the python version to 3.6+ but it's not happening. When I try conda upgrade python some upgrades and removals happen but python is still 3.5.
conda update python gives:
Collecting package metadata (current_repodata.json): done
Solving environment: \
Updating python is constricted by
anaconda -> requires python==3.8.3=hcff3b4d_2
If you are sure you want an update of your package either try `conda update --all` or install a specific version of the package you want using `conda install <pkg>=<version>`
done
# All requested packages already installed.
While using conda install python=3.7 after installing/upgrading so many packages the following error message is obtained:
Downloading and Extracting Packages
traitlets-4.3.3 | 140 KB | ##################################### | 100%
conda-token-0.3.0 | 10 KB | ##################################### | 100%
testpath-0.4.4 | 82 KB | ##################################### | 100%
click-7.1.2 | 71 KB | ##################################### | 100%
singledispatch-3.4.0 | 16 KB | ##################################### | 100%
conda-package-handli | 881 KB | ##################################### | 100%
astropy-4.0.1.post1 | 6.1 MB | ##################################### | 100%
greenlet-0.4.16 | 24 KB | ##################################### | 100%
navigator-updater-0. | 691 KB | ##################################### | 100%
defusedxml-0.6.0 | 23 KB | ##################################### | 100%
importlib-metadata-1 | 51 KB | ##################################### | 100%
ipython-7.16.1 | 987 KB | ##################################### | 100%
numba-0.50.1 | 3.1 MB | ##################################### | 100%
mkl_fft-1.1.0 | 143 KB | ##################################### | 100%
pycurl-7.43.0.5 | 68 KB | ##################################### | 100%
scikit-learn-0.23.1 | 5.0 MB | ##################################### | 100%
sympy-1.6.1 | 8.6 MB | ##################################### | 100%
matplotlib-base-3.2. | 5.4 MB | ##################################### | 100%
pandocfilters-1.4.2 | 13 KB | ##################################### | 100%
pkginfo-1.5.0.1 | 44 KB | ##################################### | 100%
ipykernel-5.3.2 | 179 KB | ##################################### | 100%
anaconda-navigator-2 | 5.1 MB | ##################################### | 100%
pylint-2.5.3 | 441 KB | ##################################### | 100%
kiwisolver-1.2.0 | 84 KB | ##################################### | 100%
setuptools-49.2.0 | 743 KB | ##################################### | 100%
bokeh-2.1.1 | 5.4 MB | ##################################### | 100%
cryptography-2.9.2 | 552 KB | ##################################### | 100%
zope.interface-4.7.1 | 204 KB | ##################################### | 100%
ujson-1.35 | 25 KB | ##################################### | 100%
_ipyw_jlab_nb_ext_co | 4 KB | ##################################### | 100%
asn1crypto-1.3.0 | 164 KB | ##################################### | 100%
pyzmq-19.0.1 | 459 KB | ##################################### | 100%
terminado-0.8.3 | 26 KB | ##################################### | 100%
conda-content-trust- | 56 KB | ##################################### | 100%
jsonschema-3.2.0 | 91 KB | ##################################### | 100%
bitarray-1.4.0 | 88 KB | ##################################### | 100%
beautifulsoup4-4.9.1 | 165 KB | ##################################### | 100%
certifi-2020.6.20 | 156 KB | ##################################### | 100%
nbconvert-5.6.1 | 451 KB | ##################################### | 100%
zope.event-4.4 | 10 KB | ##################################### | 100%
cffi-1.14.0 | 224 KB | ##################################### | 100%
path-13.1.0 | 35 KB | ##################################### | 100%
scikit-image-0.16.2 | 23.1 MB | ##################################### | 100%
python-language-serv | 81 KB | ##################################### | 100%
werkzeug-1.0.1 | 240 KB | ##################################### | 100%
conda-repo-cli-1.0.4 | 47 KB | ##################################### | 100%
anaconda-2020.07 | 17 KB | ##################################### | 100%
numpy-1.18.5 | 5 KB | ##################################### | 100%
flask-1.1.2 | 78 KB | ##################################### | 100%
psutil-5.7.0 | 319 KB | ##################################### | 100%
jinja2-2.11.2 | 103 KB | ##################################### | 100%
gevent-20.6.2 | 1.6 MB | ##################################### | 100%
six-1.15.0 | 13 KB | ##################################### | 100%
distributed-2.20.0 | 992 KB | ##################################### | 100%
wheel-0.34.2 | 51 KB | ##################################### | 100%
numpy-base-1.18.5 | 4.1 MB | ##################################### | 100%
lxml-4.5.2 | 1.2 MB | ##################################### | 100%
pip-20.1.1 | 1.7 MB | ##################################### | 100%
isort-4.3.21 | 69 KB | ##################################### | 100%
spyder-kernels-1.9.2 | 97 KB | ##################################### | 100%
pandas-1.0.5 | 7.8 MB | ##################################### | 100%
spyder-4.1.4 | 5.6 MB | ##################################### | 100%
python-3.7.7 | 45.1 MB | ##################################### | 100%
pytest-5.4.3 | 388 KB | ##################################### | 100%
backports-1.0 | 160 KB | ##################################### | 100%
conda-build-3.21.5 | 549 KB | ##################################### | 100%
sqlalchemy-1.3.18 | 1.5 MB | ##################################### | 100%
cytoolz-0.10.1 | 376 KB | ##################################### | 100%
pillow-7.2.0 | 617 KB | ##################################### | 100%
notebook-6.0.3 | 4.0 MB | ##################################### | 100%
keyring-21.2.1 | 58 KB | ##################################### | 100%
conda-4.10.3 | 2.9 MB | ##################################### | 100%
statsmodels-0.11.1 | 7.8 MB | ##################################### | 100%
decorator-4.4.2 | 14 KB | ##################################### | 100%
scipy-1.5.0 | 14.4 MB | ##################################### | 100%
llvmlite-0.33.0 | 17.3 MB | ##################################### | 100%
pywavelets-1.1.1 | 3.5 MB | ##################################### | 100%
regex-2020.6.8 | 324 KB | ##################################### | 100%
pyrsistent-0.16.0 | 94 KB | ##################################### | 100%
Preparing transaction: failed
# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<
Traceback (most recent call last):
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/exceptions.py", line 1079, in __call__
return func(*args, **kwargs)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main.py", line 84, in _main
exit_code = do_call(args, p)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/conda_argparse.py", line 83, in do_call
return getattr(module, func_name)(args, parser)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/main_install.py", line 20, in execute
install(args, parser, 'install')
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 317, in install
handle_txn(unlink_link_transaction, prefix, args, newenv)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/cli/install.py", line 346, in handle_txn
unlink_link_transaction.execute()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 245, in execute
self.verify()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/common/io.py", line 88, in decorated
return f(*args, **kwds)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 222, in verify
self.prepare()
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 211, in prepare
grps = self._prepare(self.transaction_context, stp.target_prefix,
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in _prepare
packages_info_to_link = tuple(read_package_info(prec, pcrec)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/core/link.py", line 288, in <genexpr>
packages_info_to_link = tuple(read_package_info(prec, pcrec)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 89, in read_package_info
package_metadata = read_package_metadata(epd)
File "/home/xx/anaconda3/lib/python3.8/site-packages/conda/gateways/disk/read.py", line 144, in read_package_metadata
data = json.loads(f.read())
File "/home/xx/anaconda3/lib/python3.8/json/__init__.py", line 357, in loads
return _default_decoder.decode(s)
File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/home/xx/anaconda3/lib/python3.8/json/decoder.py", line 355, in raw_decode
raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
`$ /home/xx/anaconda3/bin/conda install python=3.7`
environment variables:
CIO_TEST=<not set>
COMPIZ_BIN_PATH=/usr/bin/
CONDA_DEFAULT_ENV=base
CONDA_EXE=/home/xx/anaconda3/bin/conda
CONDA_PREFIX=/home/xx/anaconda3
CONDA_PROMPT_MODIFIER=(base)
CONDA_PYTHON_EXE=/home/xx/anaconda3/bin/python
CONDA_ROOT=/home/xx/anaconda3
CONDA_SHLVL=1
CURL_CA_BUNDLE=<not set>
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
PATH=/home/xx/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/u
sr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/home/xx/s
park-3.1.2-bin-
hadoop3.2/bin:/home/xx/anaconda3/bin:/usr/lib/jvm/java-8-openjdk-
amd64/jre/bin
PYTHONPATH=/home/xx/spark-3.1.2-bin-hadoop3.2/python:
REQUESTS_CA_BUNDLE=<not set>
SSL_CERT_FILE=<not set>
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
active environment : base
active env location : /home/xx/anaconda3
shell level : 1
user config file : /home/xx/.condarc
populated config files :
conda version : 4.10.3
conda-build version : 3.18.11
python version : 3.8.3.final.0
virtual packages : __linux=4.15.0=0
__glibc=2.23=0
__unix=0=0
__archspec=1=x86_64
base environment : /home/xx/anaconda3 (writable)
conda av data dir : /home/xx/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/xx/anaconda3/pkgs
/home/xx/.conda/pkgs
envs directories : /home/xx/anaconda3/envs
/home/xx/.conda/envs
platform : linux-64
user-agent : conda/4.10.3 requests/2.24.0 CPython/3.8.3 Linux/4.15.0-142-generic ubuntu/16.04.7 glibc/2.23
UID:GID : 1000:1000
netrc file : None
offline mode : False
An unexpected error has occurred. Conda has prepared the above report.
Can someone help me figure out the issue? How can I resolve this and upgrade the Python package?
1 answers
solution1
0 ACCPTED 2021-11-08 16:26:56
I resolved the issue by manually installing pyspark and making a minor change in the environment variables.
After downloading the required version of spark, you need to configure environment variables. There are a few Spark home paths you need to add to the user profile as follows,
echo "export SPARK_HOME=/opt/spark" >> ~/.profile
echo "export PATH=$PATH:$SPARK_HOME/bin:$SPARK_HOME/sbin" >> ~/.profile
echo "export PYSPARK_PYTHON=/usr/bin/python3" >> ~/.profile
Here, for me just after adding the spark home path and other parameters my python version downgrades to 3.5 in anaconda.
So I changed the Python path in the user profile to: PYSPARK_PYTHON=/usr/bin/python3.7
which resolved the issue since pyspark is compatible with python3.6+
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.
Related Question
Python - Issue in Installing Argpath for python 2.6.6 version
Change the python version after installing django
Installing numpy using port (Python default version issue)
installing pip on python2.7.5 and the old version issue
Version error after pip installing MySQL-python on OSX
Numpy import error even after installing it with current python version
Local python version not changing after installing pyenv-win
"No Python at..." error after installing different version and changing the path variable
Installing Local Version of Python
Installing an older version of python
Related Tags