如何在计算机上的python2和3上运行PySpark / spark？

Question

I have Spark/Pyspark for python 2.7.

I also have python 3.6 on my computer.

How can I get it spark/pyspark to automatically run on python3.6?

Spark/pyspark will only currently run on 2.7 for my computer

Answer 1

Use:

• PYSPARK_PYTHON environment variable to set Python interpreter for executors.
• PYSPARK_DRIVER_PYTHON environment variable to set Python interpret for the driver.

PYSPARK_PYTHON should be set on each worker node. It is best to use $SPARK_HOME/conf/spark-env.sh , ie

$ cat $SPARK_HOME/conf/spark-env.sh

#!/usr/bin/env bash

PYSPARK_PYTHON={which python3.6}