如何在 MapReduce 中将信息从一个减速器传递到另一个减速器

Question

I have two mappers and two reducers (1 and 2). My driver class runs two jobs that take in two different input files and output two different output files (again, 1 and 2). So job1 is associated with inputfile1, outputfile1, map1 and reduce1; similarly for job2.

Reduce1 outputs data of the form (id, number) using context.write. My goal is to access these key value pairs in Reduce2, but I cannot figure out how. I have tried doing a conf.setInt in reduce1, but the values are not being passed into reduce2, because I don't get them when I do a conf.getInt.

Answer 1

Coming back to this, in my experience, you write output to one path, then run another job with that path as input...

In theory, you can create two Jobs in one program if you add more code after the waitForCompletion method on the first Job