How to set up multiple brokers on the Kafka Cluster with jhipster

Question

I created a basic app with jhipster and added Apache Kafka. I have no problem producing and consuming even with another solution (from my app to a php client for kafka). Now, I want to create multiple brokers on the cluster, but from java not the .sh files.

I know the cluster is setup with the server.properties file where the id of the brokers, the log dir and other things are implied. But in my jhipster app the broker id is declared in the kafka.yml so I guess I have to edit the .yml files to declare another broker.

version: '2'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.2.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
      - 2181:2181
  kafka:
    image: confluentinc/cp-kafka:5.2.1
    environment:
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_BROKER_ID: 2
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
    ports:
      - 9092:9092

The goal is to have one jhipster app with kafka, creating multiple brokers in the cluster instead of one. Therefore I would have multiple topics. I don't have any results

Answer 1

With this docker-compose.yml you get a cluster with three brokers. The brokers are accessible from inside docker as kafka1:9092, kafka2:9092, kafka3:9092 and from docker host as localhost:19092,localhost:29092,localhost:39092 :

version: "3.7"
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:5.4.0
    hostname: zookeeper
    container_name: zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

  kafka1:
    image: confluentinc/cp-server:5.4.0
    hostname: kafka1
    container_name: kafka1
    depends_on:
      - zookeeper
    ports:
      - "19092:19092"
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP:PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092,PLAINTEXT_HOST://localhost:19092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

  kafka2:
    image: confluentinc/cp-server:5.4.0
    hostname: kafka2
    container_name: kafka2
    depends_on:
      - zookeeper
    ports:
      - "29092:29092"
    environment:
      KAFKA_BROKER_ID: 102
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka2:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

  kafka3:
    image: confluentinc/cp-server:5.4.0
    hostname: kafka3
    container_name: kafka3
    depends_on:
      - zookeeper
    ports:
      - "39092:39092"
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka3:9092,PLAINTEXT_HOST://localhost:39092
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 2
      KAFKA_GROUP_INITIAL_REBALANCE_DELAY_MS: 0
      KAFKA_CONFLUENT_LICENSE_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'false'

Answer 2

you can create multiple broker with confluentince/cp-kafka by adding more brokers in your docker-compose.yml file.

version: '2'
services:
   zookeeper:
    image: confluentinc/cp-zookeeper:5.2.1
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
    ports:
      - 2181:2181
  kafka-1:
    image: confluentinc/cp-kafka:latest
    hostname: kafka-1
    ports:
      - "19092:19092"
    depends_on:
      - zookeeper
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-1:19092

  kafka-2:
    image: confluentinc/cp-kafka:latest
    hostname: kafka-2
    ports:
      - "29092:29092"
    depends_on:
       - zookeeper
    environment:
      KAFKA_BROKER_ID: 2
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-2:29092

  kafka-3:
    image: confluentinc/cp-kafka:latest
    hostname: kafka-3
    ports:
      - "39092:39092"
    depends_on:
       - zookeeper
    environment:
      KAFKA_BROKER_ID: 3
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka-3:39092

Reference : https://better-coding.com/building-apache-kafka-cluster-using-docker-compose-and-virtualbox/

Answer 3

"Therefore I would have multiple topics" ==> Not sure to understand, but you don't need multiple brokers to have multiple topics, you can have multiple topics handled by only one broker.

I don't really know jhipster, but it looks like your yml file is exactly like a docker compose file, so I'll give you my 2 cents as if it was all started by docker compose.

You first need your brokers to connect to a same zookeeper cluster, should be fine from what I saw ( if you declare all your brokers in the same docker compose yml file)

You need to set your advertised listeners using the IP address your client will access, if you use localhost, they won't be able to connect to your broker:

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092

Should be something like :

KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://EXPOSEDIPADDRESS:9092

You might also add LISTENERS like : KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092

Be sure to use different broker IDs for each brokers, and use different ports for each your brokers ( if they run in the same box behind docker)

Yannick