如何在 docker-compose 導致的 kafka 容器上安裝 kafkacat

Question

我的上下文是我正在嘗試創建一個 docker-compose，它將啟動幾個容器來運行 ELK+FileBeats 和 3 個 Kafka 容器。 整個 docker compose 及其 yml 配置文件可以在github docker-compose 中找到

總而言之，這是我的 docker-compose 部分下載並啟動我的 kafka 容器之一：

  kafka1:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    links:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: "168"
      KAFKA_LOG_RETENTION_BYTES: "100000000"
      KAFKA_ZOOKEEPER_CONNECT:  zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_CREATE_TOPICS: "log:3:3"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

我想使用 kafkat 調查 kafka，我得到了

C:\Dockers\megalog-try-1>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
bash-4.4# kafkacat -b megalog-try-1_kafka1_1:9092 L
bash: kafkacat: command not found
bash-4.4# apt-get install kafkacat
bash: apt-get: command not found
bash-4.4# exit
exit

已編輯

暫定安裝kafkacat后的日志

/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
/bin/sh: kafkacat: not found
/ # apk --no-cache add ca-certificates wget
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/1) Installing wget (1.20.3-r0)
Executing busybox-1.29.3-r10.trigger
OK: 359 MiB in 76 packages
/ # wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
/ # wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
--2020-02-05 22:56:46--  https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
Resolving github.com... 18.228.52.138
Connecting to github.com|18.228.52.138|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive [following]
--2020-02-05 22:56:46--  https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.236.203
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.236.203|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22602 (22K) [application/vnd.android.package-archive]
Saving to: 'kafkacat-1.5.0-r0.apk'

kafkacat-1.5.0-r0.apk         100%[=================================================>]  22.07K  --.-KB/s    in 0.1s

2020-02-05 22:56:48 (167 KB/s) - 'kafkacat-1.5.0-r0.apk' saved [22602/22602]

/ # apk add --no-cache kafkacat-1.5.0-r0.apk
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/6) Installing lz4-libs (1.8.3-r2)
(2/6) Installing db (5.3.28-r1)
(3/6) Installing libsasl (2.1.27-r2)
(4/6) Installing librdkafka (0.11.6-r1)
(5/6) Installing yajl (2.1.0-r0)
(6/6) Installing kafkacat (1.5.0-r0)
Executing busybox-1.29.3-r10.trigger
OK: 361 MiB in 82 packages
/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafkacat -b megalog-try-1_kafka1_1:9092 L
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
/ #

根據 Cricket_007 的建議編輯

docker-compose.yml（僅編輯行）

...
image: confluentinc/cp-kafka:latest
...
image: confluentinc/cp-zookeeper:latest
...

相關日志

卡夫卡：

===> Configuring ...
advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.

動物園管理員：

ZOOKEEPER_CLIENT_PORT is required.

Command [/usr/local/bin/dub ensure ZOOKEEPER_CLIENT_PORT] FAILED !

Answer 1

你必須使用apk在這個 kafka 鏡像容器上安裝任何包。

目前這個包的安裝方式是使用wget或者curl拉入，使用apk安裝本地文件：

apk --no-cache add ca-certificates wget
wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
apk add --no-cache kafkacat-1.5.0-r0.apk

Answer 2

請改用 KAFKA_ADVERTISED_LISTENERS

只需刪除端口變量。 你已經有了這個KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092

ZOOKEEPER_CLIENT_PORT 是必需的

添加它

ZOOKEEPER_CLIENT_PORT: 2181

---

您可以混合和匹配容器...我只是建議您將現有的 kafkacat 容器與您現有的設置一起使用，它可以連接到任何遠程代理。

或者只是在本地系統上安裝 kafkacat