为什么 kafka 集群会出现错误“活动代理的数量 '0' 不符合所需的复制因子”？

Question

I have 2 kafka brokers and 1 zookeeper. Brokers config: server.properties file: 1 broker:

auto.create.topics.enable=true
broker.id=1
delete.topic.enable=true
group.initial.rebalance.delay.ms=0
listeners=PLAINTEXT://5.1.2.3:9092
log.dirs=/opt/kafka_2.12-2.1.0/logs
log.retention.check.interval.ms=300000
log.retention.hours=168
log.segment.bytes=1073741824
max.message.bytes=105906176
message.max.bytes=105906176
num.io.threads=8
num.network.threads=3
num.partitions=10
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
replica.fetch.max.bytes=105906176
socket.receive.buffer.bytes=102400
socket.request.max.bytes=105906176
socket.send.buffer.bytes=102400
transaction.state.log.min.isr=1
transaction.state.log.replication.factor=1
zookeeper.connect=5.1.3.6:2181
zookeeper.connection.timeout.ms=6000

2 broker:

auto.create.topics.enable=true
broker.id=2
delete.topic.enable=true
group.initial.rebalance.delay.ms=0
listeners=PLAINTEXT://18.4.6.6:9092
log.dirs=/opt/kafka_2.12-2.1.0/logs
log.retention.check.interval.ms=300000
log.retention.hours=168
log.segment.bytes=1073741824
max.message.bytes=105906176
message.max.bytes=105906176
num.io.threads=8
num.network.threads=3
num.partitions=10
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=1
replica.fetch.max.bytes=105906176
socket.receive.buffer.bytes=102400
socket.request.max.bytes=105906176
socket.send.buffer.bytes=102400
transaction.state.log.min.isr=1
transaction.state.log.replication.factor=1
zookeeper.connect=5.1.3.6:2181
zookeeper.connection.timeout.ms=6000

if i ask zookeeper like this:

echo dump | nc zook_IP 2181

i got:

SessionTracker dump:
Session Sets (3):
0 expire at Sun Jan 04 03:40:27 MSK 1970:
1 expire at Sun Jan 04 03:40:30 MSK 1970:
        0x1000bef9152000b
1 expire at Sun Jan 04 03:40:33 MSK 1970:
        0x1000147d4b40003
ephemeral nodes dump:
Sessions with Ephemerals (2):
0x1000147d4b40003:
        /controller
        /brokers/ids/2
0x1000bef9152000b:
        /brokers/ids/1

looke fine, but not works:(. Zookeeper see 2 brokers, but in first kafka broker we have error:

 ERROR [KafkaApi-1] Number of alive brokers '0' does not meet the required replication factor '1' for the offsets topic (configured via 'offsets.topic.replication.factor'). This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

also we use kafka_exporter for prometheus, and he log this error:

Cannot get oldest offset of topic Some.TOPIC partition 9: kafka server: Request was for a topic or partition that does not exist on this broker." source="kafka_exporter.go:296

pls help? were i mistake in config

Answer 1

Are your clocks working? Zookeeper thinks it's 1970

Sun Jan 04 03:40:27 MSK 1970

You may want to look at the rest of the logs or see if Kafka and Zookeeper are actively running and ports are open.

In your first message, after starting a fresh cluster you see this, so it's not a true error

This error can be ignored if the cluster is starting up and not all brokers are up yet. (kafka.server.KafkaApis)

The properties you show, though, have listeners on entirely different subnets and you're not using advertised.listeners

Answer 2

Kafka broker.id changes maybe cause this problem. Clean up the kafka metadata under zk, note: kafka data will be lost

Answer 3

I got this error message in this situation:

• Cluster talking in SSL
• Every broker is a container
• Updated the certificate with new password inside ALL brokers
• Rolling update

After the first broker reboot, it spammed this error message and the broker controller talked about "a new broker connected but password verification failed".

Solutions:

1. Set the new certificate password with the old password
2. Down then Up your entire cluster at once
3. (not tested yet) Change the certificate on one broker, reboot it then move to the next broker until you reboot all of them (ending with the controller)