Kinesis data stream skipping records with exception at downstream

Question

I have an application with the following set up. Kinesis data stream (retention period: 1day) ->

StreamExecutionEnvironment.getExecutionEnvironment().addSource(new FlinkKinesisConsumer).map(new MapFunction()).addSink();

When the MapFunction crashes due to some transient exception, the Flink will restart all the operators. At the restart, the record that caused the crash was lost and never retried.

The intended behavior I am looking for is to reprocess the record although it might still cause exceptions.

I understand that based on https://docs.aws.amazon.com/streams/latest/dev/troubleshooting-consumers.html#w3aac23c27b5 KCL could skip records, but in our case, the exception is raised in MapFunction not in FlinkKinesisConsumer. We cannot afford loosing any record, and if there is crash-loop, I would rather handling manually than being silently skipped at KCL.

Is there a way to configure this not to loose any record?

Answer 1

If everything is set up correctly, Flink should retry the failed record until it is removed from the stream. In case TRIM_HORIZON is used as starting stream position, the removal could be quite fast.

A minimum example of reading from file (tested with Flink 1.14.2 running from IDE):

public static void main(String[] args) throws Exception {
    StreamExecutionEnvironment env = StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(new Configuration());
    env.setParallelism(1);

    // fails immediately if next line is commented out
    // also, next line has no effect in Kinesis Data Analytics
    env.enableCheckpointing(10000, CheckpointingMode.EXACTLY_ONCE);

    final FileSource<String> source = FileSource
            .forRecordStreamFormat(new TextLineFormat(), new Path("input.txt")).build();

    env.fromSource(source, WatermarkStrategy.noWatermarks(), "file-source")
            .map(Integer::parseInt)
            .addSink(new PrintSinkFunction<>());

    env.execute("Test");
}

input.txt:

1
2
3
abc