Connecting Kafka inside Docker from another host

Question

I've built Kafka and Zookeeper inside docker in a host-A (192.168.1.200), and already set these environments

KAFKA_ADVERTISED_LISTENERS=INSIDE://kafka-broker:9093,OUTSIDE://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
KAFKA_LISTENERS=INSIDE://kafka-broker:9093,OUTSIDE://kafka-broker:9092
KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE
KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181

And then when I tried to connect from other host-B (IP 192.168.1.8) to bootstrap-server, the connection could not be established. Port 9092 already exposed to outside and I can telnet from host-B.

[root@node-B kafka]# ./bin/kafka-topics.sh --create --topic hehe --bootstrap-server 192.168.1.200:9092 --partitions 3
[2020-12-07 16:42:22,262] WARN [AdminClient clientId=adminclient-1] Connection to node 1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

But when I tried to connect from host-B to zookeeper server, it can connected and create topic as well

[root@node-B kafka]# ./bin/kafka-topics.sh -zookeeper 192.168.1.200:2181 --create --topic dummytopic --partitions 1 --replication-factor 1
Created topic dummytopic.

I also can create topic from host-A by using bootstrap server

[root@node-A kafka] #./bin/kafka-topics.sh --bootstrap-server 192.168.1.200:9092 --create --topic testopic --partitions 1 --replication-factor 1
Created topic testopic.

is there any configuration that i missed?

Answer 1

You're connecting to the OUTSIDE listener ( --bootstrap-server 192.168.1.200:9092 ) which in your KAFKA_ADVERTISED_LISTENERS will return its host address as localhost:9092 - which is what you see in this error because the client then connects to this returned host ( localhost , on which Kafka broker is not running):

Connection to node 1 (localhost/127.0.0.1:9092) could not be established.

You need to configure your listeners correctly for your topology. This article goes into all the details.