次要排序：何时对值进行排序？

Question

I have implemented secondary sorting for my requirement. But I need some clarity on the internal working of the same.

Given that sorting happens on the map side. I assume that all the (k,V) pairs in the spill files are ordered by keys. In our case, the composite keys.

I would like to know how the values belonging the same key from many mapfiles come in a specific order (As specified in the SortComparator) to reduce fucntion every single time.

If sorting happens on the map side and merging is done on the reducer side. How and when the values belonging to key from many map files are arranged in a particular order before the reduce function starts ?

Answer 1

Values are not sorted by default, only the keys. However, you can override Partitioner , SortComparator and GroupingComparator in specific way that makes hadoop framework to sort both keys and values in any way you like. (Example of such setup can be found in my article ). Beware that because typically value objects are much larger, jobs which order both keys and values will run for much longer than with only keys sorting.

Keys are sorted in both mappers and reducers:

1. mappers sort KV pairs for every reducer output resulting in that each output file is sorted according to SortComparator
2. reducer takes a lot of sorted files from mappers and merges them together, providing input to reduce() invocations

Values come to reduce() in some unspecified order by default. Generally, it will depend on everything: order in which you emit key/value pairs in map, order in which hadoop decides to merge files, sort algorithm used and so on.