使用spark / scala追加/连接两个文件

Question

I have multiple files stored in HDFS, and I need to merge them into one file using spark. However, because this operation is done frequently (every hour). I need to append those multiple files to the source file.

I found that there is the FileUtil that gives the 'copymerge' function. but it doesn't allow to append two files.

Thank you for your help

Answer 1

You can do this with two methods:

 sc.textFile("path/source", "path/file1", "path/file2").coalesce(1).saveAsTextFile("path/newSource")

Or as @Pushkr has proposed

 new UnionRDD(sc, Seq(sc.textFile("path/source"), sc.textFile("path/file1"),..)).coalesce(1).saveAsTextFile("path/newSource")

If you don't want to create a new source and overwrite the same source every hour, you can use dataframe with save mode overwrite ( How to overwrite the output directory in spark )