min.insync.replica 配置是否会影响 Kafka 生产者吞吐量？

Question

From kafka documentation

When a producer sets acks to "all" (or "-1"), this min.insync.replica configuration specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful.

It says when the minimum number of in-sync replicas acknowledge, the write is successful but when i run performance test with min.insync.replica as 1 and 3 (for a topic of partition=1 and RF=5 in 5 broker setup), the performance of kafka producer, with acks='all' , is same.

So, Does min.insync.replica per-topic configuration affects Kafka producer throughput (ran in isolation) with acks="all" ?

Answer 1

If you use acks='all' , the leader waits until in-sync replicas get the message before sending back an acknowledgment or an error, so the performance is affected. In case of min.insync.replica=1 , the producer gets a response back once the message is written to the leader. It should be faster than using min.insync.replica=3 as in this case the producer waits for 2 replicas to get all the messages before it can consider the message as committed.

Your results mean that the latency between your brokers is very low. I believe you should see the difference if you start the brokers in different datacenters/regions.

Answer 2

min.insync.replica is the minimum number of replicas that must acknowledge that data was received successfully for a write to be successful.

Throughput will be definitely affected if you set min.insync.replica to 3 and acks=all but won't be affected if you set acks=0 or 1 , but when you do this there is a possibility of data loss if the leader fails.

Answer 3

if you DO NOT set acks='all' and min.insync.replica > 1 be aware that you are risking data loss. If the leader goes down, it means there is no guarentee that the replicated node is a copy of the leading one. That was actually the main idea behind Kafka preventing such cases as a distributed system.

Answer 4

When a producer sets acks to "all" (or "-1"), min.insync.replicas specifies the minimum number of replicas that must acknowledge a write for the write to be considered successful

When used together, min.insync.replicas and acks allow you to enforce greater durability guarantees. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all"

HIgher the min.insync.replicas, the leader for that partition needs to wait for higher number of data copies to be written synchronously - hence lower performance.