I have built a kafka docker container and orchestrate it using docker-compose.
Calling docker ps
I get the following putput:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
5bde6f76246e hieutrtr/docker-kafka:0.0.1 "/start.sh" About an hour ago Up About an hour 7203/tcp, 0.0.0.0:32884->9092/tcp dockerkafka_kafka_3
be354f1b8cc0 hieutrtr/docker-ubuntu:devel "/usr/bin/supervisor About an hour ago Up About an hour 22/tcp producer1
50d3203af90e hieutrtr/docker-kafka:0.0.1 "/start.sh" About an hour ago Up About an hour 7203/tcp, 0.0.0.0:32883->9092/tcp dockerkafka_kafka_2
61b285f39615 hieutrtr/docker-kafka:0.0.1 "/start.sh" 2 hours ago Up 2 hours 7203/tcp, 0.0.0.0:32882->9092/tcp dockerkafka_kafka_1
20c9c5ccec05 jplock/zookeeper:3.4.6 "/opt/zookeeper/bin/ 2 hours ago Up 2 hours 2888/tcp, 3888/tcp, 0.0.0.0:32881->2181/tcp dockerkafka_zookeeper_1
I can run a producer and a consumer from inside the docker container, but it is not working from outside the docker network.
For example :
I run a kafka producer on my localhost and the following error appears:
$ kafka_2.9.1-0.8.2.1: bin/kafka-console-producer.sh --topic test --broker-list $DOCKER_HOST:32884
[2015-08-31 06:55:15,450] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
to
[2015-08-31 06:55:20,214] WARN Failed to send producer request with correlation id 2 to broker 1 with data for partitions [test,0] (kafka.producer.async.DefaultEventHandler)
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:100)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:73)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
This is my kafka docker example on github that includes the mentioned problem.
So, is anyone experiencing the same problems and can help me in any way?
Additional info :
(Just fork from ches/kafka and modify something for docker-compose) :
在Kafka服务器属性中,您需要将advertised.host.name
和advertised.port
设置为正在运行的容器的ip / port,然后它应该可以工作。
You need to put the name of the host machine where the docker instance was deployed. Also you need to map ports from docker host machine(public) to the docker container instance(private).
TL;DR Expose port 9092 on host and map it to 9092 container port to access kafka brokers outside container. See docker-compose documentation for details.
I think the problem is that you don't expose port 9092 outside container. According to your docker ps listing your 9092 container port is mapped dynamically to host's port range 32882-32884. When you connect to broker configured this way you receive metadata that contains port 9092 for advertising. With this metadata producer tries to do other requests through port 9092 and fails.
For the record, another way to get my local kafka consumer communicating with remote broker inside a Docker container was to add an entry in my /etc/hosts : docker-host-ip-address docker-kafka-container-hostname
Anyway Lundahl's solution worked fine for me and seems cleaner. Even cleaner would be to set advertised.listeners = host-ip : port since advertised.host.name and advertised.port are deprecated.
Here are my two cents, since I had a hard time figuring this one out.
My $KAFKA_HOME/config/server.properties contains the following:
listener.security.protocol.map=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
advertised.listeners=INSIDE://${container_ip}:9092,OUTSIDE://${outside_host_ip}:29092
listeners=INSIDE://:9092,OUTSIDE://:29092
inter.broker.listener.name=INSIDE
This is creating two connections, one to be used inside docker and another to be used outside. You have to choose a new port for the latter, in my case 29092, make sure this port is exposed and mapped by docker.
I was not yet able to figure out a solution without the ${outside_host_ip} in the environment, hence I'm providing the host machine's ip as an env var.
Test:
./kafka-topics.sh -zookeeper zookeeper:2181 --create --topic dummytopic --partitions 1 --replication-factor 1
./kafka-console-producer.sh --broker-list 0.0.0.0:29092 --topic dummytopic
and input a message I hope this helps others
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.