pyspark running from ubuntu wsl in windows, error: cmd not found

Question

I am installing pyspark in ubuntu wsl in windows 10. These are the commands I used after installing wsl from Microsoft Store.

#install Java runtime environment (JRE)
sudo apt-get install openjdk-8-jre-headless
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-amd64/jre
#download spark, visit https://spark.apache.org/downloads.html if you want a different version
wget https://apache.osuosl.org/spark/spark-2.4.7/spark-2.4.7-bin-hadoop2.7.tgz
#untar and set a symlink
sudo tar -xvzf spark-2.4.7-bin-hadoop2.7.tgz -C /opt
sudo ln -s spark-2.4.7-bin-hadoop2.7 /opt/spark
ln -s /usr/bin/python3.8.5 python
/opt/spark/bin/pyspark

Error:

---

/mnt/c/Users/akash/.pyenv/pyenv-win/bin/pyenv: 3: cmd: not found
/mnt/c/Users/akash/.pyenv/pyenv-win/bin/pyenv: 3: cmd: not found

I am not able to understand why it is looking in C drive. How to solve this? Any help is appreciated.

Answer 1

Instead of creating a symbolic link, try moving the unpacked directory spark-3.0.1-bin-hadoop2.7 to the opt/spark directory:

sudo mv spark-3.0.1-bin-hadoop2.7 /opt/spark

Also, go for the latest version which is Spark 3.0.1 with Hadoop 2.7.

You can read thru this online article for additional details.