Consumer not able to consume all the message from Producer

Question

I have created 3 separate container: 1 for Kafka, 2nd for Porducer(Streamer) and the last one for Consumer using docker-compose

    version: "3"

services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    networks:
      - stream-network
  kafka:
    image: confluentinc/cp-kafka:latest
    depends_on:
      - zookeeper
    ports:
      - 9092:9092
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
    networks: 
        - stream-network
  streamer:
    build:
      context: ./streamingProducer/
    networks: 
      - stream-network
    depends_on:
      - kafka
  consumer:
    build:
      context: ./streamingConsumer/
    networks: 
      - stream-network
    depends_on:
      - kafka

I am producing 10 messages from the producer inside a container, below is the code:

    from confluent_kafka import Producer
import pprint
from faker import Faker
#from bson.json_util import dumps
import time


def delivery_report(err, msg):
    """ Called once for each message produced to indicate delivery result.
        Triggered by poll() or flush(). """
    if err is not None:
        print('Message delivery failed: {}'.format(err))
    else:
        print('Message delivered to {} [{}]'.format(msg.topic(), msg.partition()))


# Generating fake data

myFactory = Faker()
myFactory.random.seed(5467)

for i in range(10):

    data = myFactory.name()
    print("data: ", data)

    # Produce sample message from localhost
    # producer = KafkaProducer(bootstrap_servers=['localhost:9092'], retries=5)
    # Produce message from docker
    producer = Producer({'bootstrap.servers': 'kafka:29092'})

    producer.poll(0)

    #producer.send('live-transactions', dumps(data).encode('utf-8'))
    producer.produce('mytopic', data.encode('utf-8'))

    # block until all async messages are sent
producer.flush()
    # tidy up the producer connection
    # producer.close()
time.sleep(0.5)

and this is the following 10 output messages:

**

streamer_1   | producer.py:35: DeprecationWarning: PY_SSIZE_T_CLEAN will be required for '#' formats
streamer_1   |   producer.produce('mytopic', data.encode('utf-8'))
streamer_1   | data:  Denise Reed
streamer_1   | data:  Megan Douglas
streamer_1   | data:  Philip Obrien
streamer_1   | data:  William Howell
streamer_1   | data:  Michael Williamson
streamer_1   | data:  Cheryl Jackson
streamer_1   | data:  Janet Bruce
streamer_1   | data:  Colton Martin
streamer_1   | data:  David Melton
streamer_1   | data:  Paula Ingram

**

When I am trying to consume the message by consumer it only consumes the last message which is in this case: Paula Ingram and then the programs run forever like an infinite loop. Not sure whats wrong. Here is the following code of the consumer:

from kafka.consumer import KafkaConsumer
try:
    print('Welcome to parse engine')
    # From inside a container
    #consumer = KafkaConsumer('test-topic', bootstrap_servers='kafka:29092')
    # From localhost
    consumer = KafkaConsumer('mytopic', bootstrap_servers='localhost:9092', auto_offset_reset='earliest')
    for message in consumer:
        print('im a message')
        print(message.value.decode("utf-8"))

except Exception as e:
    print(e)
    # Logs the error appropriately. 
    pass

Any help would be appreciated. Thanks.

Answer 1

I suspect you are having consumer group issues.

auto_offset_reset='earliest' only applies when a group does not already exist . If a group exists, then the group resumes from the last available position.

---

If that is not the case, it is not clear in what order you are running the consumer and producer, but I would start the consumer first, then docker-compose restart streamer a few times