Apache Beam Pipeline KafkaIO - 手动提交偏移量

Question

I have a beam pipeline to consume streaming events with multiple stages(PTransforms) to process them. See the following code,

pipeline.apply("Read Data from Stream", StreamReader.read())
        .apply("Decode event and extract relevant fields", ParDo.of(new DecodeExtractFields()))
        .apply("Deduplicate process", ParDo.of(new Deduplication()))
        .apply("Conversion, Mapping and Persisting", ParDo.of(new DataTransformer()))
        .apply("Build Kafka Message", ParDo.of(new PrepareMessage()))
        .apply("Publish", ParDo.of(new PublishMessage()))
        .apply("Commit offset", ParDo.of(new CommitOffset()));

The streaming events read by using the KafkaIO and the StreamReader.read() method implementation is like this,

public static KafkaIO.Read<String, String> read() {
    return KafkaIO.<String, String>read()
            .withBootstrapServers(Constants.BOOTSTRAP_SERVER)
            .withTopics(Constants.KAFKA_TOPICS)
            .withConsumerConfigUpdates(Constants.CONSUMER_PROPERTIES)
            .withKeyDeserializer(StringDeserializer.class)
            .withValueDeserializer(StringDeserializer.class);
}

After we read a streamed event/message through the KafkaIO, we can commit the offset. What i need to do is commit the offset manually , inside the last Commit offset PTransform when all the previous PTransforms executed.

The reason is, I am doing some conversions, mappings and persisting in the middle of the pipeline and when all the things done without failing, i need to commit the offset. By doing so, if the processing fails in the middle, i can consume same record/event again and process.

My question is, how do i commit the offset manually? Appreciate if its possible to share resources/sample codes.

Answer 1

Well, for sure, there are Read.commitOffsetsInFinalize() method, that is supposed to commit offsets while finalising the checkpoints, and AUTO_COMMIT consumer config option, that is used to auto-commit read records by Kafka consumer.

Though, in your case, it won't work and you need to do it manually by grouping the offsets of the same topic/partitiona/window and creating a new instance of Kafka client in your CommitOffset DoFn which will commit these offsets. You need to group the offsets by partition, otherwise it may be a race condition with committing the offsets of the same partition on different workers.