使用HDFS中的文件到Apache Spark

Question

I have few files in my hdfs and I want to use them in Spark. I am able to see my files when I give the following command:

bin/hadoop dfs -ls /input

How should I give the path of this file in spark to create an RDD:

val input=sc.textFile("???")

Answer 1

If your Spark installation is properly configured, then your normal HDFS paths should just work verbatim unchanged in Spark as well:

val input = sc.textFile("/input")

If that's not working then you probably need to make sure your Spark configuration is properly picking up your Hadoop conf dir .

You might also want to try directly checking your file listings from your Spark code to make sure the configurations are getting imported properly:

import org.apache.hadoop.conf.Configuration
import org.apache.hadoop.fs._

val path = new Path("/input")
path.getFileSystem(new Configuration()).listStatus(path)