如何将HDFS小文件合并为一个大文件？

Question

I have number of small files generated from Kafka stream so I like merge small files to one single file but this merge is based on the date ie the original folder may have number of previous files but I only like to merge for given date files to one single file.

Any suggestions?

Answer 1

Use something like the code below to iterate over the smaller files and aggregate them into a big one (assuming that source contains the HDFS path to your smaller files, and target is the path where you want your big result file):

val fs = FileSystem.get(spark.sparkContext.hadoopConfiguration)
fs.listStatus(new Path(source)).map(_.getPath.toUri.getPath).
foreach(name => spark.read.text(name).coalesce(1).write.mode(Append).text(target))

This example assumes text file format, but you can just as well read any Spark-supported format, and you can use different formats for source and target, as well

Answer 2

you should be able to use .repartition(1) to write all results to 1 file. if you need to split by date, consider partitionBy("your_date_value") .

if you're working within HDFS and S3, this may also be helpful. you might actually even use s3-dist-cp and stay within HDFS.

https://aws.amazon.com/blogs/big-data/seven-tips-for-using-s3distcp-on-amazon-emr-to-move-data-efficiently-between-hdfs-and-amazon-s3/#5

There's a specific option to aggregate multiple files in HDFS using a --groupBy option based na regular expression pattern. So if the date is in the file name, you can group based on that pattern.

Answer 3

You can develop a spark application. Using this application read the data from small files and create dataframe and write dataframe to big file in append mode.