How to produce/consume data from/to Docker using Kafka?

Question

I have seen a lot of questions and topics but they are all related to docker-compose and creation of kafka container. But I have a 1 namenode and 3 datanode container system. I have two ports related to docker container, one is 8088 and the other is 50070. I want to send data using kafka from my local to the docker. But for me I need to send the data using 8088 or 50070. But I can't figure out how to do this. I edited

listeners=SASL_PLAINTEXT://0.0.0.0:8088, PLAINTEXT://0.0.0.0:9092 advertised.listeners=SASL_PLAINTEXT://localhost:8088, PLAINTEXT://localhost:9092 security.inter.broker.protocol=SASL_PLAINTEXT

in server.properties but it didn't work. I am trying like this: I created a topic named test in Docker. I sent some data to it on Docker terminal. And in local I try to consume the data from topic with --bootstrap-server localhost:8088 but it gives error. Thank you.

Answer 1

have a 1 namenode and 3 datanode container system

These are not Kafka services, and Docker is irrelevant to the question.

If you want to send data to Hadoop datanodes on these ports, you need to use Kafka Connect, or from Python you can use Spark, Flink, etc.

If all services are running in containers, then Compose or your host isn't completely relevant, either.

You'd use a bridge network - https://docs.docker.com/network/bridge/

I need to send the data using 8088 or 50070.

Ports don't matter, but you really should consider leaving Kafka ports as the defaults.

But you must forward the listener port from the host, properly -p 8088:8088 , and you must set the necessary SASL properties in whatever command you're using with --bootstrap-server localhost:8088 , which should work from both the host and inside the container (which is where you should first debug the problem)