使用 apache Flink 读取键控 Kafka 记录？

Question

i'm using a value + record kafka producer using:

bin/kafka-console-producer.sh --topic test3 --property "parse.key=true" --property "key.separator=:" --bootstrap-server localhost:9092

but im finding hard to understand how to read those kafka records using Flink kafka consumer KafkaSource . I want to be able to do things like:

record.getValue(), record.getKey(), record.getTimestamp()...

this is my current code that only reads non keyed records from kafka

        KafkaSource<String> source = KafkaSource.<String>builder()
            .setBootstrapServers(ip)
            .setTopics("test3")
            .setGroupId("1")
            .setStartingOffsets(OffsetsInitializer.earliest())
            .setDeserializer(KafkaRecordDeserializationSchema.valueOnly(StringDeserializer.class))
            .build();

    DataStream<String> stream = env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kafka Source");
    stream.map((MapFunction<String, String>) value -> "Receiving from Kafka : " + value).print();

can i get an example of what im looking for?

Answer 1

You need to implement a KafkaRecordDeserializationSchema (but not valueOnly), and then in its deserialize method you'll have access to a ConsumerRecord, and you can work with its key, value, headers, etc to produce whatever type you want.

There's an example in Reading Apache Kafka® headers , which is part of the Immerok Apache Flink Cookbook. Note that while that example accesses the topic, partition, offset, and timestamp from the record's headers, it doesn't use the key, which is available as record.key() .

Note: I work for Immerok.