How to expose kafka endpoint from azure and consume it from onpremise application?

Question

I am trying to do a POC for the following :

I have hosted a container of kafka on azure. I am able to send messages to this kafka instance from a consumer, when the consumer is within azure.

Kafka configuration in docker-compose:

kafka:
     image: confluentinc/cp-kafka:latest
     depends_on:
       - zookeeper
     ports:
     - "9092:9092"
     environment:
       KAFKA_ADVERTISED_HOST_NAME: kafka
       KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
       KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://kafka:9092
       KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
       KAFKA_BROKER_ID: 1
       KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
       KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
       KAFKA_HEAP_OPTS: "-Xmx256M -Xms128M"
       KAFKA_LOG_DIRS: /var/lib/winkafka

Producer Application Configuration on azure to talk to kafka:

    "Endpoint": "kafka:9092"

The communication between kafka and consumer on within azure is happening perfectly.

Now, I want to have consumer on-premises, rather than on azure. So what I have done is just expose the port 9092 from azure vm and also updated endpoint to the following:

Producer Application Configuration on local to talk to kafka on azure:

"Endpoint": "Azure Vm's Public IP:9092"

But, it is not able to connect from my local application To Kafka on azure.

Can you please suggest What I am doing wrong here and how I will be able to connect from local application to kafka running in a container on azure (or any cloud)?

Answer 1

You need to configure your Kafka listener for the separate networks (internal Azure vs public IP). See https://rmoff.net/2018/08/02/kafka-listeners-explained/ for a detailed explanation.

Disclaimer: I wrote the blog post :)