[英]I need to create a kafka image with topics already created from docker compose
[英]I need to create a kafka image with topics already created
我有一個要求,我需要使用容器中已有的主題在本地設置 kafka。我正在使用ladoop/fast-data-dev
來執行此操作
我是如何手動操作的-
docker run -d --name landoopkafka -p 2181:2181 -p 3030:3030 -p 8081:8081 -p 8082:8082 -p 8083:8083 -p 9092:9092 -e ADV_HOST=localhost landoop/fast-data-dev
運行此命令后,我的容器已啟動並正在運行。
現在我在這個容器內的 go 到 bash 就像 docker -exec -it landopkafka ZD576914BB40C9998CCE691
並使用此命令創建主題
kafka-topics --zookeeper localhost:2181 --create --topic hello_topic --partitions 1 --replication-factor 1
我的主題已創建。
但我的要求是我需要一個 docker 文件,該文件將創建主題,我只需要運行它。
或者
我需要運行的 docker 撰寫文件
伙計們,我需要幫助,因為我對 docker 和 kafka 完全陌生
我也必須這樣做! 如果我不想使用wurstmeister圖像怎么辦? 我決定制作一個自定義腳本來完成這項工作,並在一個單獨的容器中運行這個腳本。
https://github.com/yan-khonski-it/kafka-compose
請注意,它適用於使用zookeeper的 kafka 版本。 Zookeeper 是 Kafka 的必備工具嗎?
使用所有主題和 zookeeper 啟動 kafka - docker-compose up -d
。
docker-compose.yml
# These services are kafka related. This docker-compose allows to start kafka locally quickly.
version: '2.1'
networks:
demo-network:
name: demo-network
driver: bridge
services:
zookeeper:
image: "confluentinc/cp-zookeeper:${CONFLUENT_PLATFORM_VERSION}"
container_name: zookeeper
environment:
ZOOKEEPER_CLIENT_PORT: 32181
ZOOKEEPER_TICK_TIME: 2000
ports:
- 32181:32181
hostname: zookeeper
networks:
- demo-network
kafka:
image: "confluentinc/cp-kafka:${CONFLUENT_PLATFORM_VERSION}"
container_name: kafka
hostname: kafka
ports:
- 9092:9092
- 29092:29092
environment:
KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
KAFKA_BROKER_ID: 1
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT_HOST://kafka:29092
LISTENERS: PLAINTEXT://0.0.0.0:9092
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
volumes:
- /var/run/docker.sock:/var/run/docker.sock
depends_on:
- "zookeeper"
networks:
- demo-network
# Automatically creates required kafka topics if they were not created.
kafka-topics-creator:
build:
context: kafka-topic-creator
dockerfile: Dockerfile
container_name: kafka-topics-creator
depends_on:
- zookeeper
- kafka
environment:
ZOOKEEPER_HOSTS: "zookeeper:32181"
KAFKA_TOPICS: "topic_v1 topic_v2"
networks:
- demo-network
然后我有一個目錄kafka-topics-creator 。 在這里,我有三個文件create-kafka-topics.sh
, Dockerfile
, README.md
。
Dockerfile
# It is recommened to use same version as kafka broker is used.
# So no additional images are pulled.
FROM confluentinc/cp-kafka:4.1.2
WORKDIR usr/bin
# Once it is executed, this container is not needed.
COPY create-kafka-topics.sh create-kafka-topics.sh
ENTRYPOINT ["./create-kafka-topics.sh"]
創建-kafka-topics.sh
#!/bin/bash
# Simply wait until original kafka container and zookeeper are started.
sleep 15.0s
# Parse string of kafka topics into an array
# https://stackoverflow.com/a/10586169/4587961
kafkatopicsArrayString="$KAFKA_TOPICS"
IFS=' ' read -r -a kafkaTopicsArray <<< "$kafkatopicsArrayString"
# A separate variable for zookeeper hosts.
zookeeperHostsValue=$ZOOKEEPER_HOSTS
# Create kafka topic for each topic item from split array of topics.
for newTopic in "${kafkaTopicsArray[@]}"; do
# https://kafka.apache.org/quickstart
kafka-topics --create --topic "$newTopic" --partitions 1 --replication-factor 1 --if-not-exists --zookeeper "$zookeeperHostsValue"
done
README.md - 所以其他人知道如何使用它。總是記錄你的東西 - 好建議。
# Creates kafka topics automatically.
## Parameters
`ZOOKEEPER_HOSTS` - zookeeper hosts, I used value `"zookeeper:32181"` to run it locally.
`KAFKA_TOPICS` - space separated list of kafka topics. Example, `topic_1, topic_2, topic_3`.
Note, this container should run only **after** your original kafka broker and zookeeper are running.
After this container creates topics, it is not needed anymore.
一種解決方案是檢查kafka-topics-creator
容器的日志。
docker logs kafka-topics-creator
應該打印
$ docker logs kafka-topics-creator
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v1".
WARNING: Due to limitations in metric names, topics with a period ('.') or underscore ('_') could collide. To avoid issues it is best to use either, but not both.
Created topic "topic_v2".
您應該嘗試使用支持環境變量的 wurstmeister/kafka 映像在容器啟動期間創建主題。
當然,Landoop 容器還有很多其他有用的東西,但聽起來你只想要 Kafka 而不想搞亂編輯任何 Dockerfile
另一種解決方案是在運行創建腳本的Kafka 之后啟動第二個容器,然后自行停止
您可以像這樣創建 docker-compose 文件...
version: '2'
services:
zookeeper:
image: wurstmeister/zookeeper:latest
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka:0.10.2.1
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: 127.0.0.1
KAFKA_CREATE_TOPICS: "MY_TOPIC_ONE:1:1,/
MY_TOPIC_TWO:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
將您的主題放在那里並運行docker-compose up
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.