Flink DataSet API：GroupBy无法正常工作吗？

Question

In my Flink Java program I use a GroupBy-Operator as follows:

dataSet.groupBy(new KeySelector<myObject, Tuple2<Tuple2<Integer, Integer>, Integer>>() {
    private static final long serialVersionUID = 5L;
    Tuple2<Tuple2<Integer, Integer>, Integer> groupingKey = new Tuple2<Tuple2<Integer, Integer>, Integer>();

        public Tuple2<Tuple2<Integer, Integer>, Integer> getKey(myObject s) {
            groupingKey.setField(s.getPosition(), 0);
            groupingKey.setField(s.getBand(), 1);
            return groupingKey;
        }
    })
    .reduceGroup(reduceFunction);

getPosition() returns a Tuple2<Integer, Integer> and getBand() returns an int .

I want to group my dataset on both values. If I have 6 positions and 4 bands I would like to get 24 distinct groups and use the groupReduce -function for every group independently. But currently my resulting groups seem to contain various values for the band and the position. I checked this like that in the groupReduce function:

if (this.band == null) {
    this.band = myObject.getBand();
}
if (this.band != myObject.getBand()) {
    System.out.println("The band should be " + this.band + " but is: " + myObject.getBand());

Additionally there are also values in my resulting file which indicate a problem with the grouping. Is it possible that the grouping does not work in my case? Or could this just be a consequence of another potential bug in my code?

Answer 1

I think your check in the GroupReduceFunction is not working correctly. The GroupReduceFunction.reduce() can be called several times for different groups. this.band is a member variable of your GroupReduceFunction and I assume that you do not reset this.band at the end of the reduce() method.

Hence, this.band will null only in the first call of reduce() . At the beginning of the second call this.band will be initialized and won't be set to the band of the current group. Therefore, the following check will fail.