使用 apache flink 进行数据聚合

Question

I am trying to create a real time number counter using flink data streaming api. But I am facing some problem to achieve the solution.

Example: Data Payload

{
    "room": 1,  # Room Number
    "numbers": [101, 111, 201, 211, 13, ....], # Only these numbers in output with count
    "my_number": 401  # My Current Number according to room
}

There are only 4 rooms 1, 2, 3, and 4, and my_number will vary according to room. This is stream data that I am passing to the flink.

Problem Statement: I want to count number according to room and in output want to return only arrays numbers with its count. This is same for each room.

output example:
 [
    {
        101: 2,
        111: 5,
        201: 1
        .
        .
        .
    }
 ]

Answer 1

If I understand correctly, you can do this:

dataPayloadSource.keyby("room").process(new CountNumbers()).flatMap(new MapDataPayloadToCorrespondObject())addSink(...);

// ...
public class CountNumbers extends KeyedProcessFunction<..>{
    private MapState<Integer, Integer> numberCountState;
    
    public void open(Configuration config){
        // initialize state in here
    }
  
    public void processElement(DataPayload dp){
        // for each numbers in the dp.counts, get the state value with numberCountState.get(..)
        // check it returns null, if yes, map does not have that key, initialize with 1
        // if not null, then get the current value from the map, increment by 1
        // update the mapstate
    } 
}

// ...
public class MapDataPayloadToCorrespondObject extends RichFlatMapFunction<...>{
    public void flatMap(...){
        // convert DataPayload to OutputObject
    }
}