使用本地Python运行SPARK作业

Question

I am submitting a spark job as follows :

spark-submit --conf spark.ui.port=5051 server_code.py

My python is set to my home users python :

export PYSPARK_PYTHON="$HOME/software/anaconda3/bin/python3.7"

However when I run spark like this it cannot access this python so complains. Is there any way around this? I am able to run spark jobs with the above configurations when the job doesnt involve a web UI. In the case above, I am serving out a web UI with the results.

java.io.IOException: Cannot run program "/x/software/anaconda3/bin/python3.7": error=13, Permission denied

Answer 1

I solved my problem by providing access permissions to folders from the home directory. chmod 777 dir name all the way down to chmod 777 to the actual python script