Apache Spark一次运行即可读取多个文本文件

Question

I can successfully load a text file into a DataFrame with the following Apache Spark Scala code:

val df = spark.read.text("first.txt")
  .withColumn("fileName", input_file_name())
  .withColumn("unique_id", monotonically_increasing_id())

Is there any way to provide the multiple files in the single run? Something like this:

val df = spark.read.text("first.txt,second.txt,someother.txt")
  .withColumn("fileName", input_file_name())
  .withColumn("unique_id", monotonically_increasing_id())

Right now the following code doesn't work with the following error:

Exception in thread "main" org.apache.spark.sql.AnalysisException: Path does not exist: file:first.txt,second.txt,someother.txt;
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:558)
    at org.apache.spark.sql.execution.datasources.DataSource$$anonfun$org$apache$spark$sql$execution$datasources$DataSource$$checkAndGlobPathIfNecessary$1.apply(DataSource.scala:545)

How to properly load multiple text files?

Answer 1

The function spark.read.text() have a varargs parameter, from the docs :

def text(paths: String*): DataFrame

This means that to read multiple files you only need to supply them to the function separated by commas, ie

val df = spark.read.text("first.txt", "second.txt", "someother.txt")