獲取 spring kafka 中未處理的消息計數

Question

我們正在遷移到 Kafka，我需要創建一個監控 POC 服務，該服務將定期檢查 Kafka 隊列中未處理的消息計數，並根據計數采取一些措施。 但是此服務不得讀取或處理消息，指定的消費者會這樣做，對於每個 cron，此服務只需要隊列中存在的未處理消息的計數。 到目前為止，我已經從多個示例中做到了這一點

 public void stats() throws ExecutionException, InterruptedException {
    Map<String, Object> props = new HashMap<>();
    // list of host:port pairs used for establishing the initial connections to the Kafka cluster
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, false);
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    props.put(ConsumerConfig.GROUP_ID_CONFIG, groupId);

    try (final KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
    consumer.subscribe(Arrays.asList(topicName));
        while (true) {
            Thread.sleep(1000);
            ConsumerRecords<String, String> records = consumer.poll(1000); 
            if (!records.isEmpty()) {
                System.out.println("records is not empty = " + records.count() + " " + records);
            }
            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
                Set<TopicPartition> partitions = consumer.assignment();
                //consumer.seekToBeginning(partitions);
                Map<TopicPartition, Long> offsets = consumer.endOffsets(partitions);
                for (TopicPartition partition : offsets.keySet()) {
                    OffsetAndMetadata commitOffset = consumer.committed(new TopicPartition(partition.topic(), partition.partition()));
                    Long lag = commitOffset == null ? offsets.get(partition) : offsets.get(partition) - commitOffset.offset();
                    System.out.println("lag = " + lag);
                    System.out.printf("partition %s is at %d\n", partition.topic(), offsets.get(partition));
                }
            }
        }
    }
}

代碼有時運行良好，有時給出錯誤 output，請告訴我

Answer 1

不訂閱主題； 只需創建一個具有相同組的消費者即可獲得 endOffsets。

有關示例，請參見此答案。