Not able to read event from single topic that has single partition through multiple consumers

Question

I'm a newbie to Kafka, using Kafka-node. I'm using Kafka for real-time data processing. The system has one producer and multiple consumers. I'm able to receive data if any one of the consumers is running at a time but if I run two consumers, only one of them will receive data and there is no data received by another one.

Code for kafka producer is:

const config = require('../config');
const logger = require('./logger');
const kafka = require('kafka-node'),
HighLevelProducer = kafka.HighLevelProducer,
client = new kafka.Client(`${config.kafka.host}:${config.kafka.port}`),
producer = new HighLevelProducer(client);

producer.on('ready', () => {
logger.info("Events producer to kafka is ready...");
});

producer.on('error', (err) => {
logger.error("Error while starting kafka producer:" + err.message);
});

const queueEvent = (event, callback) => {
const payloads = [
{ topic: config.kafka.queueName, messages: JSON.stringify(event, null, 
2) },
];
producer.send(payloads, (err, data) => {
if (err) {
logger.error(`Error while producing data: ${err.message}`);
callback(err);
} else {
callback(null, data);
}
});
};

module.exports = {
queueEvent
};

Configuration done for all consumers is same as shown below:

const kafka = require('kafka-node');

const logger = require('../common/logger');
const config = require('../common/config');
const eventDao = require('../models/event');
const _ = require('lodash');

const { getDeviceHierarchy } = require('../common/apiUtils');

const options = { autoCommit: true, fetchMaxWaitMs: 1000, fetchMaxBytes: 
1024 * 1024 };

const HighLevelConsumer = kafka.HighLevelConsumer,
client = new kafka.Client(`${config.kafka.host}:${config.kafka.port}`),
consumer = new HighLevelConsumer(
client,
[
{ topic: config.kafka.queueName, partition: 0 }
],
options
);

I'm using docker image of Kafka and below is the settings I've done

docker run -d -p 2181:2181 -p 3030:3030 -p 8081-8083:8081-8083 -p 9581-9585:9581-9585 -p 9092:9092 -e ADV_HOST=localhost -e DISABLE=azure-documentdb,blockchain,bloomberg,cassandra,coap,druid,elastic,elastic5,ftp,hazelcast,hbase,influxdb,jms,kudu,mongodb,mqtt,redis,rethink,voltdb,yahoo,hdfs,jdbc,elasticsearch,s3,twitter -e CONNECT_HEAP=6G -e RUNNING_SAMPLEDATA=0 -e RUNTESTS=0 landoop/fast-data-dev:latest**

Answer 1

Could you please confirm if the consumers in the multiple consumer scenario belong to the same consumer group?

If they do, then the observed behaviour is correct. Let me try and elaborate this a little bit:

1. In the scenario you've described, it sounds like both the consumers belong to the same consumer group. In this case, each consumer from the group can consume only from one partition. As we have only one partition here, the first consumer in line consumes it.

2. If we have multiple consumer groups, with multiple consumers in each group subscribing to the same topic (with one partition). In this case, multiple consumers can consume from the same partition.

I'm not familiar with the programming language you've used but I couldn't find the statement setting the " group.id " property in your Kafka consumer. Could you please try setting this in your consumer code/config?

Also, could you check and confirm the version of Kafka you're using and if there is a default value in consumer.properties file in Kafka? because from version 0.9.0.0 the group.id property value has been made mandatory and not providing this should throw an error (Check this ticket ).