Connect Python to kafka on AWS EC2 from local machine

Question

I am trying to connect my python application to kafka running on AWS EC2. I am able to connect with ec2 via terminal i check with telnet <ec2 ip> 9092 . I am able to connect via this but not able to connect with python application. Even if my python application starts with without any error with ec2 ip address, i am not able to receive any data from my kafka topic from ec2 to local machine.

When i add my pulic ip address to:

advertised.listeners=PLAINTEXT://<local ip addrss>:9092 

Debezium connector with kafka-connect won't start , but without enabling advertised.listeners it works.

How do i configure kafka and kafka-connect so that i can consume kafka topic from ec2 instance on my local machine?

Answer 1

You need to set advertised.listeners to be the EC2 Public DNS/IP, restart the broker, then open the VPC / firewall connection on the listening port.

Debezium's rest.advertised.listener property is different from Kafka broker's, and you woudn't need it set on your local machine.

Python and Kafka Connect should share the same bootstrap.server protocol

You can test your listeners better using kafkacat -L -b <bootstrap>:9092