change PATH env variable on databricks

Question

I need to update my python path so that I can use python3 on databricks.

On databricks notebook:

  %sh
  echo $PATH

  /databricks/python2/bin:/databricks/python3/bin:/usr/local/sbin

Then, I run:

  %sh
  export PATH=/databricks/python3/bin:$PATH

But, I still got:

  %sh
  echo $PATH

  /databricks/python2/bin:/databricks/python3/bin:/usr/local/sbin

How can I change PATH? Also, when I created a cluster on databricks, the python version is 3.6 but why python2 is used ?

thanks

Answer 1

According to the offical document Cluster Configurations / Databricks Runtime / Python version of Databricks, it sounds like your current databricks cluster is created for Runtime 5.5 or below with Python 2, because the databricks runtime version above 5.5 only supports Python 3, as the figure below.

So if you only want to use Python 3 in databricks, you just need to create a cluster which runtime version above 5.5.

Or if you have to use Runtime 5.5, please create a new cluster to specify Python 2. Note: I tried to set the environment variable PYSPARK_PYTHON with the Python 3 path via %sh export PYSPARK_PYTHON=.... on a existing running cluster, it do not work.