Flink 重复数据删除和 processWindowFunction

Question

i'm creating a pipeline where the inputs are json messages containing a timestamp field, used to set eventTime. The problem is about the fact that some record could arrive late or duplicate at the system, and this situations needs to be managed; to avoid duplicates I tried the following solution:

                .assignTimestampsAndWatermarks(new RecordWatermark()
                        .withTimestampAssigner(new ExtractRecordTimestamp()))
                .keyBy(new MetricGrouper())
                .window(TumblingEventTimeWindows.of(Time.seconds(60)))
                .trigger(ContinuousEventTimeTrigger.of(Time.seconds(3)))
                .process(new WindowedFilter())
                .keyBy(new MetricGrouper())
                .window(TumblingEventTimeWindows.of(Time.seconds(180)))
                .trigger(ContinuousEventTimeTrigger.of(Time.seconds(15)))
                .process(new WindowedCountDistinct())
                .map((value) -> value.toString());

where the first windowing operation is done to filter the records based on timestamp saved in a set, as follow:

public class WindowedFilter extends ProcessWindowFunction<MetricObject, MetricObject, String, TimeWindow> {
    HashSet<Long> previousRecordTimestamps = new HashSet<>();

    @Override
    public void process(String s, Context context, Iterable<MetricObject> inputs, Collector<MetricObject> out) throws Exception {
        String windowStart = DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochMilli(context.window().getStart()));
        String windowEnd = DateTimeFormatter.ISO_INSTANT.format(Instant.ofEpochMilli(context.window().getEnd()));
        log.info("window start: '{}', window end: '{}'", windowStart, windowEnd);

        Long watermark = context.currentWatermark();
        log.info(inputs.toString());
        for (MetricObject in : inputs) {
            Long recordTimestamp = in.getTimestamp().toEpochMilli();
            if (!previousRecordTimestamps.contains(recordTimestamp)) {
                log.info("timestamp not contained");
                previousRecordTimestamps.add(recordTimestamp);
                out.collect(in);
            }
        }
    }

this solution works, but I've the feeling that I'm not considering something important or it could be done in a better way.

Answer 1

One potential problem with using windows for deduplication is that the windows implemented in Flink's DataStream API are always aligned to the epoch. This means that, for example, an event occurring at 11:59:59, and a duplicate occurring at 12:00:01, will be placed into different minute-long windows.

However, in your case it appears that the duplicates you are concerned about also carry the same timestamp. In that case, what you're doing will produce correct results, so long as you're not concerned about the watermarking producing late events.

The other issue with using windows for deduplication is the latency they impose on the pipeline, and the workarounds used to minimize that latency.

This is why I prefer to implement deduplication with a RichFlatMapFunction or a KeyedProcessFunction . Something like this will perform better than a window:

private static class Event {
  public final String key;
}

public static void main(String[] args) throws Exception {
  StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
  
  env.addSource(new EventSource())
    .keyBy(e -> e.key)
    .flatMap(new Deduplicate())
    .print();
  
  env.execute();
}

public static class Deduplicate extends RichFlatMapFunction<Event, Event> {
  ValueState<Boolean> seen;

  @Override
  public void open(Configuration conf) {
    StateTtlConfig ttlConfig = StateTtlConfig
      .newBuilder(Time.minutes(1))
      .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite)
      .cleanupFullSnapshot()
      .build();
    ValueStateDescriptor<Boolean> desc = new ValueStateDescriptor<>("seen", Types.BOOLEAN);
    desc.enableTimeToLive(ttlConfig);
    seen = getRuntimeContext().getState(desc);
  }

  @Override
  public void flatMap(Event event, Collector<Event> out) throws Exception {
    if (seen.value() == null) {
      out.collect(event);
      seen.update(true);
    }
  }
}

Here the stream is being deduplicated by key , and the state involved is being automatically cleared after one minute.