在 Kafka 中，如何将 min.insync.replicas=1 设置为 __consumer_offsets_ 主题？

Question

I have one Kafka broker.

My server.properties is as follows:

offsets.topic.replication.factor=3
default.replication.factor=3
min.insync.replicas=3

And I created a topic just for test:

sh kafka-topics.sh --bootstrap-server localhost:9092 --topic test --create replication-factor 1 --config min.insync.replicas=1

The topic is created well. It describes as follows:

Topic: test PartitionCount: 8 ReplicationFactor: 1 Configs: min.insync.replicas=1, segment.bytes=10...
Topic: test Partition: 0 Leader: 1 Replicas: 1 Isr: 1
....

When my console consumer starts working, the following error occurs:

org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(1) is insufficient to satisfy the min.isr requirement of 3 for partition __ consumer_offsets-1

As I understand, I set the min.insync.replicas to 1, but it affects only data topics, not _ consumer_offsets topics. So _ consumer_offsets still needs 3 replicas as my server.properties has min.insync.replicase=3 setting.

Is my understanding correct?? if so, is there a way to set __consumer_offsets topics' min.insync.replicas in command line topic creation?

Answer 1

"Is my understanding correct??"

Yes, your understanding is correct. The configuration min.insync.replicas can be set when creating a topic. If you do not specify this configuration it will be set based on the broker-wide configuration min.insync.replicas that is provided in the server.properties .

"if so, is there a way to set __consumer_offsets topics' min.insync.replicas in command line topic creation?"

As the topic __consumer_offsets is an internal topic and it will be created automatically, the broker-wide defaults are being used. In your case you would need to change the configuration in your server.properties file:

min.insync.replicas=1

As I understand, you only have one broker in your cluster. So having default setting of 3 for the configurations offsets.topic.replication.factor , default.replication.factor , and min.insync.replicas will cause issues (like you have seen) for topics that rely on those broker-wide settings.

---

Please do not change these setting on a production cluster without really knowing the impacts. It will have an impact on the durability of your data and your ConsumerGroups and there is a good reason why the defaults are set to 3 .

Answer 2

if your min isr configuration not taking effect, here is the solution:

Note: operation below will remove all the kafka data, this solution is purely for changing the isr settings of internal topics that have already been created, you should never do it when this data cleanup operation doesn't fit into your scenario

every time you want to change replica factor and min isr, don't forgot to clear the internal topics first:

rm -rf kafka/kafka-logs/*
rm -rf kafka/logs/*
rm -rf zookeeper/zkdata/version-2/* 
rm -rf zookeeper/logs/version-2/*

you can confirm whether your config working by describe all the topics including the internal topics:

topics=(`./kafka-topics.sh --list --zookeeper $ZK_SERVER | grep -v grep  | awk '{print $1}'`)                                                                                         
for topic in ${topics[@]}
do
./kafka-topics.sh --describe --bootstrap-server $BOOTS_STRAP_SERVER --topic $topic
done