Kafka docker compose configuration only creates one partition

Question

I'm using docker-compose to setup a kafka broker the following way:

  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      command: sh -c "(sleep 15) && (kafka-topics --create --partitions 3 --bootstrap-server kafka:9092 --topic input) && (kafka-topics --create --bootstrap-server kafka:9092 --partitions 1 --topic output)"

The problem is that the command which is supposed to create the 'input' topic with 3 partitions, is only creating one partition and I don't understand why. When I go inside the container afterwards to delete the topic and create it again, using the same command, the topic then is setup correctly.

I've tried to rebuild the image several times and clearing the cache but everytime I only end up with one partition. What am I doing wrong?

Answer 1

Why don't you separate out your init script from Kafka container and create an init-kafka-container which sets up all the required topics and other configurations?

  kafka:
    image: confluentinc/cp-kafka:latest
    hostname: kafka
    container_name: kafka
    depends_on:
      - zookeeper
    ports:
      - 29092:29092
      - 9092:9092
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
  init-kafka-container:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - kafka
    entrypoint: [ '/bin/sh', '-c' ]
    command: |
      "
      # rather than giving sleep 15 use this 
      # to block init container to wait for Kafka broker to be ready  
      kafka-topics --bootstrap-server kafka:9092 --list

      # create init topics
      kafka-topics --create --partitions 3 --bootstrap-server kafka:9092 --topic input
      kafka-topics --create --bootstrap-server kafka:9092 --partitions 1 --topic output
      "

Answer 2

command isn't a valid environment variable.

It needs to be unindented, but then you're overriding the Confluent container default command, and the container, nor Kafka broker would start at all, and no topics would be made.

---

It's creating topics with one partition because that's what the default broker config is set to for automatically created ones