NameError：名称“ SparkSession”未定义

Question

I'm new to cask cdap and Hadoop environment.

I'm creating a pipeline and I want to use a PySpark Program. I have all the script of the spark program and it works when I test it by command like, insted it doesn't if I try to copy- paste it in a cdap pipeline.

It gives me an error in the logs:

NameError: name 'SparkSession' is not defined

My script starts in this way:

from pyspark.sql import *

spark = SparkSession.builder.getOrCreate()
from pyspark.sql.functions import trim, to_date, year, month
sc= SparkContext()

How can I fix it?

Answer 1

Spark connects with the local running spark cluster through SparkContext . A better explanation can be found here https://stackoverflow.com/a/24996767/5671433 .

To initialise a SparkSession , a SparkContext has to be initialized. One way to do that is to write a function that initializes all your contexts and a spark session.

def init_spark(app_name, master_config):
    """
    :params app_name: Name of the app
    :params master_config: eg. local[4]
    :returns SparkContext, SQLContext, SparkSession:
    """
    conf = (SparkConf().setAppName(app_name).setMaster(master_config))

    sc = SparkContext(conf=conf)
    sc.setLogLevel("ERROR")
    sql_ctx = SQLContext(sc)
    spark = SparkSession(sc)

    return (sc, sql_ctx, spark)

This can then be called as

sc, sql_ctx, spark = init_spark("App_name", "local[4]")