How to chain mapper/reducer in Hadoop 1.0.4?

Question

I was using "new" API of Hadoop 1.0.4 (classes in package org.apache.hadoop.mapreduce). When I wanted to chain mapper/reducer, I found out that ChainMapper, ChainReducer are written for the "old" API (classes in package org.apache.hadoop.mapred). What should I do?

Answer 1

I was also searching for the same. I did get the answer and even though its late I thought sharing this may help someone.

From Hadoop 2.0 onwards you can find ChainMapper and ChainReducer in the package org.apache.hadoop.mapreduce.lib.chain

ChainMapper usage pattern:

 ... Job job = new Job(conf, "MyJob"); Configuration map1Conf = new Configuration(false); ... ChainMapper.addMapper(job, AMap.class, LongWritable.class, Text.class, Text.class, Text.class, true, map1Conf); Configuration map2Conf = new Configuration(false); ... ChainMapper.addMapper(job, BMap.class, Text.class, Text.class, LongWritable.class, Text.class, false, map2Conf); Configuration map3Conf = new Configuration(false); ... ChainReducer.setReducer(job, CReducer.class, Text.class, Text.class, LongWritable.class, Text.class, false, map3Conf); ... job.waitForComplettion(true); ... 

Answer 2

Please read this post . This shows how to make use of two JobConf to enable Chaining of Map Reduce Jobs instead of using ChainMapper/ChainReducer.