CSV 格式未在 spark-shell 中加载

Question

Using spark 1.6 I tried following code:

val diamonds = spark.read.format("csv").option("header", "true").option("inferSchema", "true").load("/got_own/com_sep_fil.csv")

which caused the error

error: not found: value spark

Answer 1

In Spark 1.6 shell you get sc of type SparkContext , not spark of type SparkSession , if you want to get that functionlity you will need to instantiate a SqlContext

import org.apache.spark.sql._
val spark = new SQLContext(sc)

Answer 2

sqlContext is implict object SQL contect which can be used to load csv file and use com.databricks.spark.csv for mentionin csv file format

val df = sqlContext.read.format("csv").option("header", "true").option("inferSchema", "true").load("data.csv")

Answer 3

You need to initialize instance using SQLContext(spark version<2.0) or SparkSession(spark version>=2.0) to use methods provided by Spark.

To initialize spark instance for spark version < 2.0 use:

import org.apache.spark.sql._
val spark = new SQLContext(sc)

To initialize spark instance for spark version >= 2.0 use:

val spark = new SparkConf().setAppName("SparkSessionExample").setMaster("local")

To read the csv using spark 1.6 and databricks spark-csv package: val df = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("data.csv")