How to read first N kafka messages in flink?

Question

I'm using flink to build a pipeline whose source is kafka. For testing, I just want to read the first N messages from kafka, and need to stop the stream after that.

How can I do that? I'm using FlinkKafkaConsumer08 .

Answer 1

To do anything stateful with Flink you should be using Flink's managed state, so that your application will be fault tolerant. But if you're willing to overlook that requirement, this could be as simple as this:

public class Example {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
        env.setParallelism(1);

        env.addSource(...)
           .filter(new Limiter())
           .print();

        env.execute();
    }

    public static class Limiter implements FilterFunction<Event> {
        private transient int count = 0;

        @Override
        public boolean filter(Event e) throws Exception {
            if (++count <= 10) {
                return true;
            } else {
                return false;
            }
        }
    }
}