kafka producer docker unable to connect to broker docker - AWS

Question

Here is the yml file which is used to bring up docker containers in an AWS instance for kafka and zookeeper:

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
    - "2181:2181"
  kafka:
    build: .
    ports:
    - "9092:9092"
  environment:
    KAFKA_ADVERTISED_HOSTNAME: <machines private ip>
    KAFKA_LISTENERS: PLAINTEXT://:9092
    KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://<machines private ip>:9092


    KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock

When I run the docker-compose command with the above file, it leads to the creation a docker network called kafka-docker with a kafka container and a zookeeper container.

Now, in the default bridge docker network, I have another container with the following piece of nodejs code:

const Producer = kafka.Producer;
const client = new kafka.Client("<machines private ip>:2181");

const producer = new Producer(client);
const kafka_topic = 'hello-topic';

event = ...
event_payload = ...
let payloads = [{topic:kafka_topic ,messages:JSON.stringify(event_payload),  partition: 0 }]
let push_status = producer.send(payloads, (err, data) => {
       if (err) {
            console.log(err);
       } else {
            console.log('[kafka-producer -> '+kafka_topic+']: broker update success');
            }
        });

The console.log(err) gives me the error 'Broker not available'. Can someone please tell me what is wrong with my setup?

Answer 1

Notice the line:

const client = new kafka.Client("<machines private ip>:2181");

This is not the port that Kafka is listening on. Kafka is listening for connections on port 9092:

const client = new kafka.Client("<machines private ip>:9092");

It should work after this alteration.