Docker (Compose) client connects to Kafka too early

Question

I am trying to run Kafka with Docker and Docker Compose. This is the docker-compose.yml :

version: "2"

services:
  zookeeper:
    image: "wurstmeister/zookeeper"
    ports:
      - "2181:2181"

  kafka:
    build:
      context: "./services/kafka"
      dockerfile: "Dockerfile"
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: "0.0.0.0"
      KAFKA_CREATE_TOPICS: "test:1:1"
      KAFKA_ZOOKEEPER_CONNECT: "zookeeper:2181"
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"

  users:
    build:
      context: "./services/users"
      dockerfile: "Dockerfile"
    ports:
      - "4001:4001"
    environment:
      NODE_ENV: "develop"
      ZOOKEEPER_HOST: "zookeeper"
      ZOOKEEPER_PORT: "2181"
    volumes:
      - "./services/users:/service"

The users service only tries to connect (using kafka-node in Node.js) and listens on a topic and publishes one message to it every time it is ran.

The problem is that I keep getting Connection Refused errors. I am using Dockerize to wait for the kafka port to be available in the Dockerfile with the line CMD dockerize -wait tcp://kafka:9092 node /service/index.js .

It waits for the port to be available before starting the users container and this system works, but it is not at the right time. It seems that Kafka is opening the 9092 port before it has elected a leader.

When I run Kafka first and let it start completely and then run my app, it runs smoothly.

How do I wait for the correct moment before starting my service?

Answer 1

Try the docker-compose version 2.1 or 3, as it includes an healthcheck directive.
See " Docker Compose wait for container X before starting Y " as an example.

You can:

depends_on:
  kafka:
    condition: service_healthy

And in kafka add:

healthcheck:
    test: ["CMD", ...]
    interval: 30s
    timeout: 10s
    retries: 5

with a curl command for instance which would test if kafka has elected a leader.

Answer 2

A full example; this is what I use in docker compose.

tldr; use a kafka healthcheck
["CMD", "kafka-topics.sh", "--list", "--zookeeper", "zookeeper:2181"]

• integration test app depends on kafka
• app depends on kafka
• kafka depends on zookeeper

Since the integration test and the app are starting at the same time, I think this helps with total execution time.
Also, both are starting after kafka's healthcheck is passing.

version: '2.1'

services:

  my-integration-tests:
    image: golang:1.16
    volumes:
      - myapp:/app
    command: go test -tags=integration -mod=vendor -cover -v --ginkgo.v --ginkgo.progress --ginkgo.failFast
    depends_on:
      kafka:
        condition: service_healthy

  my-app:
    image: local/my-app
    build:
      context: .
    depends_on:
      kafka:
        condition: service_healthy

  zookeeper:
    image: wurstmeister/zookeeper:3.4.6
    expose:
      - "2181"
    tmpfs:
      - /opt/zookeeper-3.4.6/data

  kafka:
    image: wurstmeister/kafka:latest
    depends_on:
      - zookeeper
    expose:
      - 9092
    tmpfs:
      - /kafka
    environment:
      KAFKA_ADVERTISED_LISTENERS: INSIDE://localhost:9094,OUTSIDE://kafka:9092
      KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      LOG4J_LOGGER_KAFKA_AUTHORIZER_LOGGER: DEBUG, authorizerAppender
    healthcheck:
      test: ["CMD", "kafka-topics.sh", "--list", "--zookeeper", "zookeeper:2181"]
      interval: 5s
      timeout: 10s
      retries: 5