Installing pySpark gives this error : ERROR: Command errored out with exit status 1

Question

I am using Anaconda's Spyder Python 3.7 on Windows 10

I have Java

C:\Users\GX>java -version
java version "1.8.0_251"
Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)

When I try to install pySpark on Anaconda

I get this error

pip install pyspark
Collecting pyspark
  Using cached pyspark-2.4.5.tar.gz (217.8 MB)
Note: you may need to restart the kernel to use updated packages.
    ERROR: Command errored out with exit status 1:
     command: 'C:\Users\GX\anaconda3\pythonw.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\\Users\\GX\\AppData\\Local\\Temp\\pip-install-qtv58ism\\pyspark\\setup.py'"'"'; __file__='"'"'C:\\Users\\GX\\AppData\\Local\\Temp\\pip-install-qtv58ism\\pyspark\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\GX\AppData\Local\Temp\pip-install-qtv58ism\pyspark\pip-egg-info'
         cwd: C:\Users\GX\AppData\Local\Temp\pip-install-qtv58ism\pyspark\
    Complete output (31 lines):
    Could not import pypandoc - required to package PySpark
    Traceback (most recent call last):
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\installer.py", line 128, in fetch_build_egg
        subprocess.check_call(cmd)
      File "C:\Users\GX\anaconda3\lib\subprocess.py", line 363, in check_call
        raise CalledProcessError(retcode, cmd)
    subprocess.CalledProcessError: Command '['C:\\Users\\GX\\anaconda3\\pythonw.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'C:\\Users\\GX\\AppData\\Local\\Temp\\tmppc02y1qc', '--quiet', 'pypandoc']' returned non-zero exit status 2.

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "C:\Users\GX\AppData\Local\Temp\pip-install-qtv58ism\pyspark\setup.py", line 224, in <module>
        'Programming Language :: Python :: Implementation :: PyPy']
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\__init__.py", line 143, in setup
        _install_setup_requires(attrs)
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\__init__.py", line 138, in _install_setup_requires
        dist.fetch_build_eggs(dist.setup_requires)
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\dist.py", line 721, in fetch_build_eggs
        replace_conflicting=True,
      File "C:\Users\GX\anaconda3\lib\site-packages\pkg_resources\__init__.py", line 783, in resolve
        replace_conflicting=replace_conflicting
      File "C:\Users\GX\anaconda3\lib\site-packages\pkg_resources\__init__.py", line 1066, in best_match
        return self.obtain(req, installer)
      File "C:\Users\GX\anaconda3\lib\site-packages\pkg_resources\__init__.py", line 1078, in obtain
        return installer(requirement)
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\dist.py", line 777, in fetch_build_egg
        return fetch_build_egg(self, req)
      File "C:\Users\GX\anaconda3\lib\site-packages\setuptools\installer.py", line 130, in fetch_build_egg
        raise DistutilsError(str(e))
    distutils.errors.DistutilsError: Command '['C:\\Users\\GX\\anaconda3\\pythonw.exe', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', 'C:\\Users\\GX\\AppData\\Local\\Temp\\tmppc02y1qc', '--quiet', 'pypandoc']' returned non-zero exit status 2.
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

runfile('C:/Users/GX/OneDrive/__PhD/POC/Pred04d.py', wdir='C:/Users/GX/OneDrive/__PhD/POC')

I tried using pip3 but that did not work.

Any idea how to fix that?

Answer 1

I found the solution

I used

conda install pyspark

that works