Issues with kafka connect datagen while running confluent platform on windows docker toolbox

Question

I am trying to run a bare minimum confluent community example on docker for windows (toolbox) using the example given here:

https://docs.confluent.io/current/quickstart/cos-docker-quickstart.html

but seems like all components gets started only ksql-datagen is failing of them does not work,

$ docker-compose.exe ps
The system cannot find the path specified.
     Name                    Command               State                         Ports
------------------------------------------------------------------------------------------------------------
broker            /etc/confluent/docker/run        Up       0.0.0.0:29092->29092/tcp, 0.0.0.0:9092->9092/tcp
connect           bash -c if [ ! -d /usr/sha ...   Up       0.0.0.0:8083->8083/tcp, 9092/tcp
control-center    /etc/confluent/docker/run        Up       0.0.0.0:9021->9021/tcp
ksql-cli          /bin/sh                          Up
ksql-datagen      bash -c echo Waiting for K ...   Exit 1
ksql-server       /etc/confluent/docker/run        Up       0.0.0.0:8088->8088/tcp
rest-proxy        /etc/confluent/docker/run        Up       0.0.0.0:8082->8082/tcp
schema-registry   /etc/confluent/docker/run        Up       0.0.0.0:8081->8081/tcp
zookeeper         /etc/confluent/docker/run        Up       0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

I tried looking at the logs

$ docker logs ksql-datagen

and got this exception, seems like it is trying to use a different IP address then what it is using currently

[kafka-admin-client-thread | adminclient-1] WARN org.apache.kafka.clients.NetworkClient - [AdminClient clientId=adminclient-1] Connection to node -1 (broker/172.18.0.3:29092) could not be established. Broker may not be available.

I suspect the url should be 192.168.99.100 if we are using windows toolbox and it should be running in the windows host network.

Looking forward for help understanding what why the default setup is not working, do we need any change in docker-compose.yaml file.

Answer 1

TL;DR - your problem seem to be that ksql-datagen cant find your kafka broker. you probably need to set the broker's advertised.listeners property to http://broker:9092 (or KAFKA_ADVERTISED_LISTENERS for kafka property with docker) (see advertised.listeners here - https://docs.confluent.io/current/installation/configuration/broker-configs.html )

Long answer - every broker (maybe other modules as well) expose their 'address' in that property - that way clients can find them. they can expose a local docker address - to be accessed by clients running on the same docker network (eg on address kafka0:9092 where kafka0 is an internal docker alias), a localhost - to be accessed by clients running on the same host but outside network (eg localhost:9092), or they can expose a real IP of the host (to be accessed by clients from the outside world, eg blabla.com:9092).

For a full detailed and thorough explanation, i would recommend this great article written by a developer @ confluent - https://rmoff.net/2018/08/02/kafka-listeners-explained/