减速器可以将消息传递给Hadoop mapreduce中的驱动程序吗？

Question

I have to implement a loop of map-reduce jobs. Each iteration will terminate or continue depending on the previous one. The choice to be made is based on "is one word appears in the reducer output".

Of course I can inspect the whole output txt file with my driver program. But it is just a single word and going through the whole file will overkill. I am considering is there any way to build the communication between reducer and the driver, the reducer can notify the driver once it detects the word? Since the message to be transferred is few.

Answer 1

Your solution will be not a clean solution and hard to maintain.

There are multiple ways to achieve what you have asked for .

 1. Reducer as soon as it finds a word writes to a HDFS location (opens file on hdfs predefine filedir and writes there)
 2. client keeps polling the predefined filedir / output dir of the job. If the output dir is found and there is no filedir it means word wasnt there. 
 3. Use Zookeper

Best solution would be to , emit from mapper only if the word is found, else not emit anything. This will fasten your job and spawn a single reducer. Now you can safely check if the output of the job has any file on output or not. Use Lazy initialization, in case no rows comes to reducer no output file would be created