如何从火花数据帧中的 AWS S3 读取多个文件？

Question

I have a bunch of files in S3 bucket with this pattern

myfile_2018_(0).tab
myfile_2018_(1).tab
myfile_2018_(2).tab
..
myfile_2018_(150).tab

I would like to create a single Spark Dataframe by reading all these files. How do I create this regular expression pattern and read the files?

The files have headers. I am using Scala to do achieve this task.

Answer 1

只需指定一个 glob 模式如下（假设它们在同一个文件夹中）：

val input = spark.read.option("header", true).csv("/path/to/data/myfile_2018_([0-9]*).tab")

Answer 2

input = spark.read
.format('com.databricks.spark.csv')
.option("delimiter", "\001")
.option("header", "true")
.option("nullValue", "null")
.load("s3://bucket-name/path/to/data/myfile_2018_([0-9]*).tab", schema=in_schema)

in_schema - you can pass your own schema if you want to or else remove this option.

You can directly read from the folder above if you need all the files present in the folder and the schema is same.

input = spark.read
.format('com.databricks.spark.csv')
.option("delimiter", "\001")
.option("header", "true")
.option("nullValue", "null")
.load("s3://bucket-name/path/to/data/")