控制hadoop映射器输出文件的数量

Question

I have a job for hadoop. When the job is stated, i have some number of mappers started. And each mapper write some file to disk, like part-m-00000, part-m-00001. As I understand, each mapper create one part file. I have big amount of data, so there must be more than one mapper, but can I somehow control number of this output files? I mean, hadoop will start, for example 10 mappers, but there will be only three part files?

I found this post How do multiple reducers output only one part-file in Hadoop? But there is using old version of hadoop library. I'm using classes from org.apache.hadoop.mapreduce.* and not from org.apache.hadoop.mapred.*

I'm using hadoop version 0.20, and hadoop-core:1.2.0.jar

Is there any possibility to do this, using new hadoop API?

Answer 1

The number of output files equals to the number of reducers or the number of the mappers if there aren't any reducers.

You can add a single reducer to your job so that the output from all the mappers will be directed to it and your get a single output file. Note that will be less efficient as all the data (output of mappers) will be sent over the wire (network IO) to the node where the reducer will run. Also since a single process will (eventually) get all the data it would probably run slower.

By the wat,the fact that there are multiple parts shouldn't be very significant as you can pass the directory containing them to subsequent jobs

Answer 2

Im not sure you can do it (your link is about multiple outputs not converging to only one), and why use only one output ? you will lose all parallelism on sort ?

Im also working on big files (~10GB each) and my MR process almost 100GB each. So to lower Map numbers, I set a higher value of block size in hdfs (applies only to newer files) and a higher value of mapred.min.split.size in mapred-site.xml

Answer 3

You might want to look at MultipleOutputFormat

Part of what Javadoc says:

This abstract class extends the FileOutputFormat, allowing to write the output data to different output files.

Both Mapper and Reducer can use this.

Check this link for how you can specify a output file name or more from different mappers to output to HDFS.

NOTE: And, moreover, make sure you don't use context.write() so that, 10 files from 10 mapper don't get created. Use only MultipleOutputFormat to output.

Answer 4

If the job has no reducers, partitioners and combiners, each mapper outputs one output file. At some point, you should run some post processing to collect the outputs into large file.