PySpark：无法运行程序“/home/user/venv/bin/python”：错误=2，没有这样的文件或目录

Question

I'am trying to create a virtual env and start a PySpark-shell using it. Here my code:

export PATH=/appl/python/3.6.6/bin:$PATH #location of python on the linux server
python3.6 -m venv ./venv1
source venv1/bin/activate
python -m pip install --index-url http://host/repository/pypi-all/simple --trusted-host host pandas
deactivate

(download packages from a local repository)

Now I start the PySpark-Shell with this:

pyspark --conf spark.pyspark.virtualenv.bin.path=/home/user/venv1 --conf spark.pyspark.python=/home/user/venv1/bin/python

It works but i can't run function that use sparkcontext or spark session. For exemple, i'am tryting to use sc.parallelize() or spark.createDataFrame() it bring me this error:

java.io.IOException: Cannot run program "/home/user/venv1/bin/python": error=2, No such file or directory

Notice that i'am on a server on my entreprise, so i can only access to my folder in home.

May be the problem is /home/user/venv/bin/python is a symlink. So i retry with this command:

python3.6 -m venv --copies ./venv1

This replace the symlink by the real file or folder. But same error.

How is it possible that PySpark bring me this error while everything looks good, no?

Answer 1

I don't really understand why I have this error but I found a solution. My code works well with spark-sumbit but I want the PySpark-Shell.

My solution is to export these two variables:

export PYSPARK_DRIVER_PYTHON=./venv1/bin/python
export PYSPARK_PYTHON=/path/python/3.6.6/bin/python3.6

and execute this line:

pyspark --conf spark.yarn.appMasterEnv.PYSPARK_PYTHON=./venv1/bin/python --master yarn --deploy-mode client --archives ./venv1.zip#VENV

It need that you have created a zip of your venv with:

zip -rq ../venv1.zip *