How to know howmany data consumed from kafka queue and what are data existing inside kafka queue topic?

Question

I am producing data and consuming data in "topicDemo" Kafka to enable distributed data and process those data parallel.

But in real time scenario need to monitor currently how much data in queue( topicDemo ) and how much data consumed from ( topicDemo ).

Is there any kafka API available to give those details?

This is code I am producing data

  // create instance for properties to access producer configs
        Properties props = new Properties();

        // props.put("serializer.class",
        // "kafka.serializer.StringEncoder");
        props.put("bootstrap.servers", "localhost:9092");
        props.put("metadata.broker.list", "localhost:9092");

        props.put("producer.type", "async");
        props.put("batch.size", "500");
        props.put("compression.codec", "1");
        props.put("compression.topic", "topicdemo");
        // props.put("key.serializer",
        // "org.apache.kafka.common.serialization.StringSerializer");
        props.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

        org.apache.kafka.clients.producer.Producer<Integer, byte[]> producer = new KafkaProducer<Integer, byte[]>(
                props);

            producer.send(new ProducerRecord<Integer, byte[]>("topicdemo", resume.getBytes()));
        producer.close();

This is code that I am consuming data

  String topicName = "topicDemo";

    Properties props = new Properties();

    props.put("bootstrap.servers", "localhost:9092");
    props.put("group.id", "test");
    props.put("enable.auto.commit", "true");
    props.put("auto.commit.interval.ms", "1000");
    props.put("session.timeout.ms", "30000");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

    KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);

    // Kafka Consumer subscribes list of topics here.
    consumer.subscribe(Arrays.asList(topicName));

    try {
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(5);
            for (ConsumerRecord<String, String> record : records) {
                Consumer cons = new Consumer();
                if (cons.SaveDetails(record.value()) == 1) {
                    consumer.commitSync();
                }
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        consumer.close();
    }

Answer 1

Execute the command below to check how many messages you produced for each partition:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list host:port --topic topic_name --time -1

Execute the command below to check how many messages the consumer have consumed and lagged behind:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group (for old consumer)

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group --new-consumer (for new consumer)