我如何在我的 kafka 流应用程序中测试 Exactly Once Semantics

Question

i have a Kafka Streams DSL application, we have a requirement on exactly once processing, for the same i have added the configuration

streamConfig.put(processing.gurantee, "exactly_once");

I am using kafka 2.7 I have 2 queries

1. what's the difference between exactly_once and exactly_once_beta
2. how do i test this functionality to be sure my messages are getting processed only once

Thanks!

Answer 1

exactly_once_beta is an improvement over exactly_once . While exactly_once uses a transactional producer for each stream task (combination of sub-topology and input partition, exactly_once_beta uses a transactional producer for each stream thread of a Kafka Streams client. Every producer comes with separate memory buffers, a separate thread, separate network connections which might limit scaling the number of input partitions (ie number of tasks). A high number of producers might also cause more load on the brokers. Hence, exactly_once_beta has better scaling characteristics. You can find more details in KIP-447 .

Note that exactly_once will be deprecated and exactly_once_beta will be renamed to exactly_once_v2 in Apache Kafka 3.0. See KIP-732 for more details.

For tests you can get inspiration from the tests in the Apache Kafka repo:

• https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/EosIntegrationTest.java
• https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/integration/EOSUncleanShutdownIntegrationTest.java
• https://github.com/apache/kafka/blob/trunk/tests/kafkatest/tests/streams/streams_eos_test.py

Basically, you need to create a failover scenario and verify that messages are not produced multiple times to the output topics. Note that messages may be processed multiple times, but the results in the output topics must appear as if they were only processed once. You can find a pretty good talk about exactly-once semantics that also explains the failover scenarios here: https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/