有关调试Hadoop MapReduce作业中的辅助排序问题的任何技巧？

Question

I believe (believed?) I understand how secondary sort works in Hadoop. I created an intermediate key consisting of 4 fields. I partition by the first field, group by the first and second, and sort by all 4.

It looks like I nailed grouping and partitioning down, but the values come into reducer out of order.

Any ideas on how to approach debugging of this?

Answer 1

At the moment, it appears that static code review either manually or using tools works good. I believe I broke the rule: when overriding compareTo() , don't forget to override equals() and hashCode() . I'll keep everyone posted if fixing this solved the problem.