如何使用循环在Spark-Scala中迭代HDFS中的多个文本文件？

Question

I'm working in a cluster. I need to run the same spark operation for each text file contained in HDFS. But I want to do that without submitting spark job shell-command for each file from shell command line, because the number of files is 90. How can I do that?

My code for one file is structured as follow:

object SparkGraphGen{
def main(args: Array[String]){
      val conf = new SparkConf()
                .setMaster("yarn")
                .setAppName("dataset")
      val sc = new SparkContext(conf)
      val sqlContext = new org.apache.spark.sql.SQLContext(sc)
      import sqlContext.implicits._
      val peopleRDD = sc.textFile("file1.csv")
      ...
      do stuff
      ...
      sc.stop()
      }}

Answer 1

update:

1. how about foreach loop:

    val sc = new SparkContext(conf) //val files = new File("Data\\\\files\\\\").listFiles.map(_.getAbsolutePath).toList val files = new File("Data\\\\files\\\\").listFiles.map(_.getName).toList files.foreach { file => //val lines = sc.textFile(file) val lines = sc.textFile("Data\\\\files\\\\" + file) println("total lines in file " + file + " " + lines.count()) //do more stuf... for each file lines.saveAsTextFile("Data\\\\output\\\\" + file + "_output") } sc.stop() 

   output:

    total lines in file C:\\Users\\rpatel\\workspaces\\Spark\\Data\\files\\file1.txt 4 total lines in file C:\\Users\\rpatel\\workspaces\\Spark\\Data\\files\\file2.txt 4 

---

2. you can also write same for loop in shell-script

    #!/bin/bash for file in $(hadoop fs -ls /hdfs/path/to/files/|awk -F '|' '{print $NF}') do #run spark for each file spark-submit <options> $file /path/output/$file done 

or process all files in one shot....

you can put all files in one directory and pass only full directory path to spark context, spark will process all files in that directory:

val peopleRDD = sc.textFile("/path/to/csv_files/")

you can also combine RDDs like:

    val file1RDD = sc.textFile("file1.csv") 
    val file2RDD = sc.textFile("file2.csv")
    val allFileRDD = file1RDD ++ file2RDD // ++ nRDD

but with 90 files, I would put all files in one directory and use directory path to process all in one job...