Scala：如何在数据框中合并多个CSV文件

Question

I am writing the below code to get the csv file in RDD, I want to union multiple csv files and want to store in the single RDD variable. I am able to store the data of one csv file in RDD kindly help me how to union multiple csv files and to store in single RDD variable .

val Rdd = spark.sparkContext.textFile(“File1.csv").map(_.split(","))

I am expecting something like

 val Rdd = spark.sparkContext.textFile(“File1.csv").map(_.split(",")) union spark.sparkContext.textFile(“File2.csv").map(_.split(",")) 

Answer 1

If you have a large number of files I would suggest

val rdd = List("file1", "file2", "file3", "file4", "file5")
  .map(spark.sparkContext.textFile(_))
  .reduce(_ union _)

Or if you only know you have 0 or more files:

val rdd = getListOfFilenames()
  .map(spark.sparkContext.textFile(_))
  .foldLeft(spark.sparkContext.emptyRDD[String])(_ union _)