Pyspark 中 SparkSession 的导入错误

Question

I have version 2.0 of Spark installed. I am using Pyspark, on Python 2.7. I seem to have no difficulties creating a SparkContext, but for some reason I am unable to import the SparkSession. Does anyone know what I am doing wrong?

import pyspark
import pyspark.sql
from pyspark.sql import SparkSession
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
 ImportError: cannot import name SparkSession

Answer 1

Oddly enough this worked perfectly from a different directory. Running the files from this path did not result in an error!

/Users/.../spark-2.1.0-bin-hadoop2.7/python/

Answer 2

SparkSession 是在 Apache Spark 2 中引入的。要使用它，您应该在运行 pyspark 之前指定正确的 spark 版本：

export SPARK_MAJOR_VERSION=2

Answer 3

export the correct spark version of spark installed by you, it worked for me for my version 2.3

export SPARK_VERSION=2.3