如何为Docker容器订购Kafka启动Shell脚本？

Question

I am trying to containerize a Kafka server. A normal start-up sequence for a Kafka server is like this:

A: start Zookeeper server
B: start Broker server
C: create topic

Item A and B are long running process. And C need to wait for B to come up and running.

So I wrote a Dockerfile, with ENTRYPOINT executing a shell script for the above sequence:

#!/bin/sh

$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties &
$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties &

$KAFKA_HOME/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test_topic

However, I encountered three problems when run the docker image built from this:

• Docker requires the ENTRYPOINT process to be long-running, while above script does not. (it exits as long as the topic-creation is done)
• The broker server statement (the 2nd one) is long-running process. Currently I have to put it as background process using ending & , because otherwise the statements after it are not executed at all. (But making it background also has problem: the topic creation is executed immediately, while the broker server is not ready yet.)
• I could not put the broker server statement as the last long-running process, because the topic creation statement has to come after the server creation.

What could be a good way to arrange this start-up sequence?

Answer 1

Docker Compose covers a lot of orchestration tasks that are non trivial to implement in scripts.

The depends_on and healthcheck service configurations can be used to create proper service dependencies where service B waits for service A to be "healthy" before running.

Although Compose doesn't really have a defined concept of short running tasks yet, topic creation can always run at startup so an additional service that immediately exits works fine.

There's a compose definition, Dockerfile and check script on github that implements this dependency setup.

version: "2.1"

services:

  zookeeper:
    image: deployable/kafka:latest
    command: zookeeper
    ports:
     - "2181:2181"
    healthcheck:
      test: [ "CMD", "/kafka/check.sh", "zookeeper" ]
      interval: 30s
      timeout: 5s
      retries: 3

  kafka:
    image: deployable/kafka:latest
    command: kafka
    environment:
      ADVERTISE_LISTENERS: 'localhost:9092'
    ports:
     - "9092:9092"
    depends_on:
      zookeeper:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "/kafka/check.sh", "kafka" ]
      interval: 30s
      timeout: 5s
      retries: 3


  kafka-setup:
    image: deployable/kafka:latest
    command: setup
    depends_on:
      kafka:
        condition: service_healthy
    environment:
      KAFKA_TOPIC: my-test-topic

Answer 2

Basically you want to start ZK, then Kafka. Then somehow wait until Kafka is ready (that's the tricky part), do your job with kafka (eg topic creation in your case), and then wait until Kafka & ZK have finished (what happens on interrupt).

start-zookeeper &
ZK_PID=$!
start-kafka &
KAFKA_PID=$!

# that's the tricky part
wait_for_kafka
create-topic.sh

wait "${KAFKA_PID}"
wait "${ZK_PID}"

---

As mentioned, the Kafka-readiness might be tricky - the following ways might be helpful:

• waiting until Kafka responds to read requests (eg probing with kafka-topic.sh --list periodically)
• creating a pocket consumer / AdminClient (java kafka 0.11+) and getting metadata (similar to above point)
• checking existence of JMX beans for logs/controller etc.
• checking listening port availability

Answer 3

I suggest to wrap topic creation into separate script that is doing pause before trying to create topic, and run Kafka server not in background. Something like:

start-zookeeper &
create-topic.sh &
start-kafka

And create-topic.sh will look as following:

sleep 5s
kafka-topics --create...

PS Although instead of sleep it's better to probe Kafka's availability via nc -z