Can producers write to Kafka topic during recovery?

Question

What happens to Kafka producers when one of the replicas fail and partitions from that replica are being reassigned to other brokers? Will they start to fail? Or this failover process completely transparent?

Answer 1

The process is transparent as long as a new leader can be assigned for the failing partition leaders within a configurable period of time.

Messages assigned to partitions whose leader is in a broker that failed are queued in producer for request.timeout.ms. If a new leader is assigned within this time, which is the normal case when you have more than one broker and a RF > 1, then all queued messages will be sent to the new leader. If a new leader for failed partitions can't be assigned after that time then producer will throw org.apache.kafka.common.errors.TimeoutException: Batch Expired.

It could also happen that the producer fill the send buffer (with configurable size buffer.memory) before a new leader is available. In that case, producer will block for max.block.ms and after that will also throw a TimeoutException.

Answer 2

It depends on your producer acknowledge( acks ) and retries config, and broker side min.insync.replicas config, and topic replication-factor .

If you have a proper config and decent cluster size, one replica failover is transparent in most cases. for example, acks=1 and retry 2 or 3 times, and min.insync.replicas less than topic replication factor.