在Hadoop中没有Reducer的组合器

Question

Can I write a Hadoop code that has only Mappers and Combiners (ie mini-reducers with no reducer)?

job.setMapperClass(WordCountMapper.class);
job.setCombinerClass(WordCountReducer.class);

conf.setInt("mapred.reduce.tasks", 0);

I was trying to do so but I always see that I have one reduce task on the job tracker link

Launched reduce tasks = 1

How can I delete reducers while keeping combiners? is that possible?

Answer 1

You need to tell your job that you don't care about the reducer: JobConf.html#setNumReduceTasks(int)

// new Hadoop API
jobConf.setNumReduceTasks(0);

// old Hadoop API
job.setNumReduceTasks(0);

You can achieve the something with IdentityReducer .

Performs no reduction, writing all input values directly to the output.

I'm not sure whether you can keep combiners but I will start with the previous lines.

Answer 2

In the case you describe you should use Reducers. Use as key: Context.getInputSplit().getPath() + Context.getInputSplit().getStart() - this combination is unique for each Mapper.