从 scala shell 开始火花

Question

How can I start spark from an sbt shell. I don't want to use the spark-shell command. I would like to use spark and use the objects in my sbt project.

Answer 1

• Add spark dependencies to build.sbt :

libraryDependencies += "org.apache.spark" %% "spark-core" % "2.1.1",
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.1.1",

• Run sbt console:

sbt console

• Load spark session/context:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()
import spark.implicits._
val sc = spark.sparkContext

Or automate the next command with an alias:

initialCommands in console := s"""
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions._
val spark = SparkSession.builder().master("local").appName("spark-shell").getOrCreate()
import spark.implicits._
val sc = spark.sparkContext
"""