Fault tolerance in Kafka Cluster using Spring Boot

Question

I started work with Spring Boot Kafka and kafka cluster on docker. I've faced the problem when I started to check fault tolerance. Generally I created one topic (replication factor = 2) through Kafka Admin in Spring Boot. I was able to store some messages and consume them from Kafka. But after stopping one of the kafka container (I have two kafka instances and one zookeeper) my consumer was not able to communicate which remaining one.

Already I provide to bootstrap.servers property:

SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092,kafka2:9093

My kafka admin bean which create topic looks like:

@Bean
public NewTopic adviceTopic() {
    return new NewTopic(topicName, 1, (short) 2);
}

When I'm using only one instance (it's doesn't matter which one I choose) it works well (so it's not case of wrong configuration kafka listeners/host). When I'm removing one instance my application is not able to use the second one kafka instance.

My docker compose looks like:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
    - "2181:2181"
  kafka:
    container_name: kafka
    image: wurstmeister/kafka
    expose:
    - "9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://kafka:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
  kafka2:
    container_name: kafka2
    image: wurstmeister/kafka
    expose:
      - "9093"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://kafka2:9093
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_AUTO_CREATE_TOPICS_ENABLE:
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    depends_on:
      - zookeeper
  kafka-service:
     container_name: kafka_service
     build: .
     ports:
     - "8080:8080"
     environment:
       SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092,kafka2:9093
     depends_on:
       - zookeeper
       - kafka
       - kafka2

The result I've got when I used "docker stop kafka_container":

2019-07-19 09:19:16.836 WARN 1 --- [| consumers] org.apache.kafka.clients.NetworkClient : [Consumer clientId=json-0, groupId=qpm-consumers] Connection to node 1002 could not be established. Broker may not be available.

And after that I was not able to produce any messages using my Spring Boot Example. Could you provide me some tips, why my app was not able to use second one kafka instance? Even if I set properly replication factor?

Answer 1

If you have a topic with replication factor 2, the you need at least two nodes running or kafka will complain. To test you may need say 5 nodes (a,b,c,d,e) then set a topic with replication factor of 2 and check which nodes it is using then kill one of it.

Answer 2

I found the solution. I need to add to my docker-compose for kafka: KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2 KAFKA_DEFAULT_REPLICATION_FACTOR: 2

Problem was related which __consumer_offsets topic, which has 1 replication factor. Beacuse I'm using two kafka instances, I had to set these properties to 2.

Detailed answer for this question could be found here:

Kafka - Broker: Group coordinator not available