如何使用 Spark 将 S3 中的 parquet 文件组合成一个 parquet 文件

Question

I have 12 parquet files, each file represent monthly New York Taxi pick up and drop information and consist of +500K rows. I want to combine all these 12 files by row into 1 parquet file and save it in S3 to do machine learning model. How I can do that using pyspark I will upload these 12 files into AWS S3 files names

Answer 1

you can do something like, if all files are in same dir:

val ds = spark.read.parquet("/path/*").coalesce(1)
ds.write.parquet("/path/single")

or

val ds1 = spark.read.parquet("/path1/file")
val ds2 = spark.read.parquet("/path2/anotherlocation/file")
val ds = ds1.union(ds2)
ds.coalesce(1).write.parquet("/path/single")

That is an example using Scala, you can do the same in Java/Python.