永久设置findspark.init（）

Question

I have Apache Spark installed on ubuntu at this path /home/mymachine/spark-2.1.0-bin-hadoop2.7 so I have to go to python directory, located under this directory, to be able using spark OR I can use it outside python directory with help from a library called findspark, however it seems I have to always init this library like this:

import findspark
findspark.init("/home/mymachine/spark-2.1.0-bin-hadoop2.7")

everytime I want to use findspark , which is not very effective. Is there anyway to init this library permanently?

At here it mentioned need to set a variable SPARK_HOME on .bash_profile and I did it, but no luck.

Answer 1

Add the following variables to your .bashrc file

export SPARK_HOME=/path/2/spark/folder
export PATH=$SPARK_HOME/bin:$PATH

then source .bashrc
If you wish run to pyspark with jupyter notebook, add these variables to .bashrc

export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'

again source .bashrc
Now if you run pyspark from shell, it will launch jupyter notebook server and pyspark will be availble on python kernels.