如何禁用hadoop组合器？

Question

In wordcount example, the combiner is explicitly set in

job.setCombinerClass(IntSumReducer.class);

I would like to disable the combiner so that the output of mapper is not processed by the combiner. Is there a way to do that using MR config files (ie without modifying and recompiling the wordcount code)?

Thanks

Answer 1

Suppose this is your command line

hadoop jar your_hadoop_job.jar your_mr_driver \
command_line_arg1 command_line_arg2 command_line_arg3 \
-libjars all_your_dependency_jars

Here following parameters

• command_line_arg1
• command_line_arg2
• command_line_arg3

will be passed on to your main method as arg[0], arg[1] and arg[3] respectively. Assuming arg[0] and arg[1] is used for identifying input and output folder. You can use arg[3] to pass a boolean flag like ('1' or 'true' or 'yes') to understand if you want to use combiner and accordingly set combiner. Example below (default...it won't set combiner class)

if ( "YyesTrue1".contains(arg[3])){
    job.setCombinerClass(IntSumReducer.class);
}