Hadoop将多个零件文件合并为一个文件

Question

Currently I have

part-00001 part-00002

I know that using hdfs -getmerge is the best way to combine those files into a single one. However, is it possible to do it programmatically ?

I've tried using MultipleOutput , but it is not working. I've also tried writing my own CustomOutputFormat however due to multiple reducers during writing it to the file in parallel it gives org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException error when closing the Dataoutputstream.

Answer 1

You can always use the FileSystem class from your java code, and probably calling the concat method is all you need.

MultipleOutput does almost the opposite. Instead of having part-xxxxx files, it also produces customly named files, which typically means more files than before.

CustomOuputFormat is also not a good idea, since in any case, you will have as many output files as the number of reducers. The output format will not change that.

Using a single reducer ( setNumReduceTasks(1) ) could be a working solution, but unnecessarily expensive, since it "kills" parallelism (all the data are processed by a single task). Consider using it only if your data is rather small, otherwise avoid it.

Another solution would be to simply call hdfs -getmerge as a shell command from your java code, after the MapReduce job is complete.

Answer 2

You cannot do it programmatically as its managed by Hadoop and these files are created depends on the number of reducers configured . Why do you need to merge these files programatically ? If for input as another job , you can always mention the directory as input and use CombineInputFormat if there are lot of small part- files . Otherwise hdfs -getmerge is the best option if you want to merge your own .