如何知道kafka队列消耗了多少数据以及kafka队列主题中存在哪些数据？

Question

I am producing data and consuming data in "topicDemo" Kafka to enable distributed data and process those data parallel.

But in real time scenario need to monitor currently how much data in queue( topicDemo ) and how much data consumed from ( topicDemo ).

Is there any kafka API available to give those details?

This is code I am producing data

  // create instance for properties to access producer configs
        Properties props = new Properties();

        // props.put("serializer.class",
        // "kafka.serializer.StringEncoder");
        props.put("bootstrap.servers", "localhost:9092");
        props.put("metadata.broker.list", "localhost:9092");

        props.put("producer.type", "async");
        props.put("batch.size", "500");
        props.put("compression.codec", "1");
        props.put("compression.topic", "topicdemo");
        // props.put("key.serializer",
        // "org.apache.kafka.common.serialization.StringSerializer");
        props.put("key.serializer", "org.apache.kafka.common.serialization.IntegerSerializer");
        props.put("value.serializer", "org.apache.kafka.common.serialization.ByteArraySerializer");

        org.apache.kafka.clients.producer.Producer<Integer, byte[]> producer = new KafkaProducer<Integer, byte[]>(
                props);

            producer.send(new ProducerRecord<Integer, byte[]>("topicdemo", resume.getBytes()));
        producer.close();

This is code that I am consuming data

  String topicName = "topicDemo";

    Properties props = new Properties();

    props.put("bootstrap.servers", "localhost:9092");
    props.put("group.id", "test");
    props.put("enable.auto.commit", "true");
    props.put("auto.commit.interval.ms", "1000");
    props.put("session.timeout.ms", "30000");
    props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
    props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

    KafkaConsumer<String, String> consumer = new KafkaConsumer<String, String>(props);

    // Kafka Consumer subscribes list of topics here.
    consumer.subscribe(Arrays.asList(topicName));

    try {
        while (true) {
            ConsumerRecords<String, String> records = consumer.poll(5);
            for (ConsumerRecord<String, String> record : records) {
                Consumer cons = new Consumer();
                if (cons.SaveDetails(record.value()) == 1) {
                    consumer.commitSync();
                }
            }
        }
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        consumer.close();
    }

Answer 1

Execute the command below to check how many messages you produced for each partition:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list host:port --topic topic_name --time -1

Execute the command below to check how many messages the consumer have consumed and lagged behind:

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group (for old consumer)

bin/kafka-consumer-groups.sh --bootstrap-server broker1:9092 --describe --group test-consumer-group --new-consumer (for new consumer)