Anaconda + Spark - changing python version for ipython notebooks

Question

I installed Anaconda, and try to work with spark on top. When I launch spark with Ipython_OPTS="notebook", the python version is the one associated with anaconda's version of python for the notebook.

$ conda search python
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ....
ipython                      0.13                     py26_0  defaults
                          *  4.1.2                    py35_1  defaults
ipython-notebook             0.13.2                   py27_0  defaults
                             4.0.4                    py27_0  defaults
                             4.0.4                    py34_0  defaults
                             4.0.4                    py35_0  defaults
python                       1.0.1                         0  defaults
                          .  2.7.11                        0  defaults
                          *  3.5.1                         0  defaults

And if start spark-shell I can precise the environment version on which I am interested (I want 2.7.11) :

$ PYSPARK_PYTHON=/Applications/anaconda/anaconda/envs/vingt-sept/bin/python  pyspark
Python 2.7.11 |Continuum Analytics, Inc.| (default, Dec  6 2015, 18:57:58)

but if I start spark with the ipython notebook then it defaults back to the python 3.5 version :-(

How can I link the default ipython version to the same version as the one of my env "vingt-sept" ?

Answer 1

Similar to how you are setting the PYSPARK_PYTHON environment variable for the pyspark shell, you can also set this environment variable in your IPython/Jupyter notebook using:

import os
os.environ["PYSPARK_PYTHON"] = "/Applications/anaconda/anaconda/envs/vingt-sept/bin/python"

Refer to this blog post for more information about setting PYSPARK_PYTHON and other Spark-related environment variables from your notebook.