Kafka和Docker：将Kafka消息推送到另一个Dockerized Consumer

Question

Scenario: I need to push, from my Dockerized Kafka container, a message to the physical machines of my teammates.

One teammate (we'll call him Alfred), has his kafka directly installed on Windows.

The other (we'll call him Bob), is also using a Dockerized Kafka container.

Results Thus Far:

We've been able to push a kafka-produced message from my kafka contain into the consumer of a NON-DOCKERIZED consumer instance of Al's computer. ?So we know that no further configuration; producer-side; is required. ?Accessing the dockerized kafka of Rob Reris is proving trickier. ?Upon utilizing the command docker network inspect bridge we note that dockerized containers are encapsulated from the general IPv4 address of the company's network.

We identified and attempted to test a number of potential issues.

?1. Port Forwarding Rules of the receiver's VM. Adjustments to the Port Forwarding Rules have been attempted. No solution. 2. Reconfiguring the server.properties located under opt/kafka_2.11-0.9.0.0/config/ on the reciever's folder. No solution 3. Reconfiguring the server.properties located under the kafka container /etc/kafka/ No solution 4. Add an extra Line to the docker-compose.yml file of the kafka operational build we are utilizing, and rebuild kafka Bob's container. Line Added: KAFKA_LISTENERS: "PLAINTEXT://0.0.0.0-9092"

Thus far we are still not receiving when I send a test message to Bob's internal network IP.

My test message:

/bin/kafka-console-producer.sh --broker-list {Bob's IPv4}:9092 --topic test

Could it be that I actually need to target his Dockerized IP...somehow? Though I'm certain we already have that covered with the Site-to-Site Port Forwarding of VituralBox.

Assistance appreciated.

Answer 1

The problem you may be facing is the well known issue related to Kafka advertising a private IP to Zookeeper that is not reachable by clients.

When you dockerize Kafka you have two broad options:

• Launch Kafka containers using network mode = host so Kafka use host network stack.
• Use container port forwarding and make Kafka advertise the host IP and port being used setting Kafka config "advertised.listeners" to the right value. In order to do this automatically you may need to create a bootstrap script that finds out the host IP (and port) and set Kafka properties file before running the service. Take a look at wurstmeister/kafka-docker .