sc.textFile Scala错误

Question

I am trying to read a file from a path that I've written a file at. I am following the documentation so I am using the following line of code:

 val data = sc.textFile(PATH)

This give a compilation error: not found: value sc.

I found solutions that involve commands that should be given in terminal, but in my case I am using eclipse. Is there a way to resolve this issue by importing a library or something like this?

thank you!

Answer 1

You need to create the spark context first.

For example:

val conf = new SparkConf().setAppName(appName).setMaster(master)
val sc = new SparkContext(conf)

When you are using the Spark shell there is no need to create the spark context, because it is already created for you, in the variable called sc.

You can read more about the spark context initialization there .