Kafka Consumer on host machine not accessing messages of Kafka Producer running in docker

Question

I have this consumer code written on the host machine

from kafka import KafkaConsumer
KAFKA_HOSTS = 'divolte-kafka:9092'
KAFKA_VERSION = (0,11,5)

topic = "csptest"
consumer = KafkaConsumer(topic, bootstrap_servers=KAFKA_HOSTS, api_version=KAFKA_VERSION)
for msg in consumer:
    print(msg)

Kafka is installed in the docker with this configuration

version: "3.3"
services:

  # Kafka/Zookeeper container
  divolte-kafka:
    image: krisgeus/docker-kafka
    container_name: divolte-kafka
    environment:
      ADVERTISED_HOST: divolte-kafka
      KAFKA_ADVERTISED_HOST_NAME: 192.168.65.0
      LOG_RETENTION_HOURS: 1
      AUTO_CREATE_TOPICS: "false"
      KAFKA_CREATE_TOPICS: divolte:4:1
      ADVERTISED_LISTENERS: OUTSIDE://divolte-kafka:9092,INTERNAL://localhost:9093
      LISTENERS: OUTSIDE://0.0.0.0:9092,INTERNAL://0.0.0.0:9093
      SECURITY_PROTOCOL_MAP: OUTSIDE:PLAINTEXT,INTERNAL:PLAINTEXT
      INTER_BROKER: INTERNAL
    ports:
      - 9092:9092 # kafka broker
    expose:
      - "9092"
    networks:
      - divolte.io

when I try running producer and consumer as follows, it works. BUT When I start the producer and access the topic "csptest" in the consumer code written in python in the host machine, I get no message (Nothing gets printed). Thanks for helping me out.

./kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 2 --topic csptest

# producer
./kafka-console-producer.sh --broker-list localhost:9092 --topic csptest
> dd
> hi
> jhj


# consumer 
./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic csptest --group topic_group
> dd
> hi
> jhj

Answer 1

Your shell scripts work while you're in the container, but that's not going to help you test code running outside of Docker

You seem to have the internal and external listeners flipped

From the host, you want to connect on the internal listener, so you need to expose port 9093 and connect on localhost:9093, not use the Kafka container service name