HADOOP组合器操作功能

Question

I have a doubt about combiner functioning in Hadoop Map/Reduce Framework. The combiner operation is applied only on key-value pairs output by a map task or on all map tasks occurring on a given node. In fact, i have done some tests and it seems to be the first one. If I'm right, according to you, why this behavior has been chosen knowing that combining all map tasks outputs can be very beneficial to decrease bandwidth use.

thanks in advance

Answer 1

• How does it know when all the map tasks will be complete? The TaskTracker doesn't know how the JobTracker will assign map tasks. You would probably have to wait for all the map tasks to be complete before running the combiners.
• You still want to keep the data flow between mappers and reducers moving. As combiners run and output is created, that data starts getting shuffled to the reducers right away (barring slowstart configuration set to something high). This is good because it spreads out the network utilization over time.