Issues with __consumer_offsets topic in Kafka 0.10.2.0

Question

I had a single node kafka setup. It was working fine. I then added another broker and made a cluster of kafka with two nodes. I have not installed separate zookeeper and using the same zookeeper which comes with kafka package. I did below changes in order to make cluster.

Changes in zookeeper.properties in both the nodes:

dataDir=/tmp/zookeeper
clientPort=2181
maxClientCnxns=1000
server.1=10.20.40.120:2888:3888
server.2=10.20.40.119:2888:3888
initLimit=10
syncLimit=5
tickTime=2000

Copied server.properties to broker1.properties in first node and broker2.properties in second node:

Contents of broker1.properties:

broker.id=1
listeners=PLAINTEXT://10.20.40.120:9092
advertised.listeners=PLAINTEXT://10.20.40.120:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000
replica.fetch.max.bytes=4000012
message.max.bytes=2690123replica.fetch.max.bytes=4000012
message.max.bytes=2690123
max.message.bytes=4000012

Contents of broker2.properties in second node:

broker.id=2
listeners=PLAINTEXT://10.20.40.119:9092
advertised.listeners=PLAINTEXT://10.20.40.119:9092
log.dirs=/tmp/kafka-logs
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
zookeeper.connect=10.20.40.120:2181,10.20.40.119:2181
zookeeper.connection.timeout.ms=6000

Created a file myid in broker1 and put 1 in it. 2 in broker2.

Started zookeeper in both using below command:

nohup bin/zookeeper-server-start.sh config/zookeeper.properties &

Started kafka in broker1 and broker2 with below commands respectively:

nohup bin/kafka-server-start.sh config/broker1.properties &
nohup bin/kafka-server-start.sh config/broker2.properties &

Now, when I try to describe __consumer_offsets, I see below:

Topic:__consumer_offsets        PartitionCount:50       ReplicationFactor:1     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: __consumer_offsets       Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 3    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 4    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 5    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 6    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 7    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 8    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 9    Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 10   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 11   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 12   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 13   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 14   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 15   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 16   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 17   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 18   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 19   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 20   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 21   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 22   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 23   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 24   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 25   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 26   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 27   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 28   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 29   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 30   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 31   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 32   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 33   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 34   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 35   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 36   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 37   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 38   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 39   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 40   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 41   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 42   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 43   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 44   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 45   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 46   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 47   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 48   Leader: 0       Replicas: 0     Isr: 0
        Topic: __consumer_offsets       Partition: 49   Leader: 0       Replicas: 0     Isr: 0

Shouldn't the replication actor be set as 2 and Leader be 1 and 2 for different partitions and Replicas and Isr as 1,2?

When I am trying to start a consumer, I am getting COORDINATOR NOT AVAILABLE and error_code=15 . Because of this error, I found there might be some problem with my __consumer_offsets and hence with cluster.

Where is the missing link and how to correct the same?

Answer 1

You should set num.partitions to 2 in the broker properties file. Another thing to note is to you must ensure there are at least two running brokers before Kafka internally creates this topic.