How to install kafkacat on kafka container resulted from docker-compose

Question

My context is I am trying to create a docker-compose wich will start few containers for running ELK+FileBeats and 3 Kafka containers. The whole docker compose and its yml configuration files can be found in github docker-compose

To sumarize, this is my docker-compose part downloading and starting one of my kafka containers:

  kafka1:
    image: wurstmeister/kafka
    command: [start-kafka.sh]
    depends_on:
      - zoo1
      - zoo2
      - zoo3
    links:
      - zoo1
      - zoo2
      - zoo3
    ports:
      - "9092:9092"
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092
      KAFKA_BROKER_ID: 1
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_LOG_RETENTION_HOURS: "168"
      KAFKA_LOG_RETENTION_BYTES: "100000000"
      KAFKA_ZOOKEEPER_CONNECT:  zoo1:2181,zoo2:2181,zoo3:2181
      KAFKA_CREATE_TOPICS: "log:3:3"
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'

I want investigate kafka using kafkat and I got

C:\Dockers\megalog-try-1>docker exec -it megalog-try-1_kafka1_1 bash
bash-4.4# kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
bash-4.4# kafkacat -b megalog-try-1_kafka1_1:9092 L
bash: kafkacat: command not found
bash-4.4# apt-get install kafkacat
bash: apt-get: command not found
bash-4.4# exit
exit

edited

Logs after tentative to install kafkacat

/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
/bin/sh: kafkacat: not found
/ # apk --no-cache add ca-certificates wget
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/1) Installing wget (1.20.3-r0)
Executing busybox-1.29.3-r10.trigger
OK: 359 MiB in 76 packages
/ # wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
/ # wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
--2020-02-05 22:56:46--  https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
Resolving github.com... 18.228.52.138
Connecting to github.com|18.228.52.138|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive [following]
--2020-02-05 22:56:46--  https://github-production-release-asset-2e65be.s3.amazonaws.com/120518176/95d10080-eee3-11e9-8e83-c0ba853d6714?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20200206%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200206T131952Z&X-Amz-Expires=300&X-Amz-Signature=19d4778753313a7a535c3f8c10645e13851d482f49035903de83ba93c087a247&X-Amz-SignedHeaders=host&actor_id=0&response-content-disposition=attachment%3B%20filename%3Dkafkacat-1.5.0-r0.apk&response-content-type=application%2Fvnd.android.package-archive
Resolving github-production-release-asset-2e65be.s3.amazonaws.com... 52.216.236.203
Connecting to github-production-release-asset-2e65be.s3.amazonaws.com|52.216.236.203|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22602 (22K) [application/vnd.android.package-archive]
Saving to: 'kafkacat-1.5.0-r0.apk'

kafkacat-1.5.0-r0.apk         100%[=================================================>]  22.07K  --.-KB/s    in 0.1s

2020-02-05 22:56:48 (167 KB/s) - 'kafkacat-1.5.0-r0.apk' saved [22602/22602]

/ # apk add --no-cache kafkacat-1.5.0-r0.apk
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.9/community/x86_64/APKINDEX.tar.gz
(1/6) Installing lz4-libs (1.8.3-r2)
(2/6) Installing db (5.3.28-r1)
(3/6) Installing libsasl (2.1.27-r2)
(4/6) Installing librdkafka (0.11.6-r1)
(5/6) Installing yajl (2.1.0-r0)
(6/6) Installing kafkacat (1.5.0-r0)
Executing busybox-1.29.3-r10.trigger
OK: 361 MiB in 82 packages
/ # kafkacat -C -b localhost:9092 -t log -o beginning -f '\nKey (%K bytes): %k\t\nValue (%S bytes): %s\n\Partition: %p\t
Offset: %o\n--\n'
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafkacat -b megalog-try-1_kafka1_1:9092 L
Error relocating /usr/bin/kafkacat: rd_kafka_fatal_error: symbol not found
/ # kafka-topics.sh --list --bootstrap-server localhost:9092
__consumer_offsets
log
/ #

edited after Cricket_007's suggestion

docker-compose.yml (only lines edited)

...
image: confluentinc/cp-kafka:latest
...
image: confluentinc/cp-zookeeper:latest
...

relevant logs

kafka:

===> Configuring ...
advertised.port is deprecated. Please use KAFKA_ADVERTISED_LISTENERS instead.

zookeeper:

ZOOKEEPER_CLIENT_PORT is required.

Command [/usr/local/bin/dub ensure ZOOKEEPER_CLIENT_PORT] FAILED !

Answer 1

You have to use apk for installing any package on this kafka image container.

The current installation method for this package is to pull it in using wget or curl and install the local file with apk:

apk --no-cache add ca-certificates wget
wget --quiet --output-document=/etc/apk/keys/sgerrand.rsa.pub https://alpine-pkgs.sgerrand.com/sgerrand.rsa.pub
wget https://github.com/sgerrand/alpine-pkg-kafkacat/releases/download/1.5.0-r0/kafkacat-1.5.0-r0.apk
apk add --no-cache kafkacat-1.5.0-r0.apk

Answer 2

Please use KAFKA_ADVERTISED_LISTENERS instead

Just remove the port variable. You already have this KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka1:9092

ZOOKEEPER_CLIENT_PORT is required

Add it

ZOOKEEPER_CLIENT_PORT: 2181

---

You can mix and match containers... I was only suggesting that you use the existing kafkacat container with your existing setup, which could connect to any remote broker.

Or just install kafkacat on your local system