如何以编程方式获取 Java 中 Kafka 主题的最新偏移量

Question

Here is what I am trying

Collection <TopicPartition> partitions = consumer.partitionsFor(topic).stream();

And also how to indicate you've hit the end or there isn't anymore messages to consume. If the offset doesn't match the broker's end offset at the time how to do that.

Any suggestions.

Answer 1

In order to get the latest offset you can either use the command line:

./bin/kafka-run-class.sh kafka.tools.GetOffsetShell \
    --broker-list localhost:9092 \
    --topic topicName

or programmatically in Java:

KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProperties)) {
    consumer.subscribe(Arrays.asList("topicName"));
        Set<TopicPartition> assignment;
        while ((assignment = consumer.assignment()).isEmpty()) {
            consumer.poll(Duration.ofMillis(500));
        }
        consumer.endOffsets(assignment).forEach((partition, offset) -> System.out.println(partition + ": " + offset));
}

---

Now if you want to force the consumer to start consuming from the latest offset, you can either use the following property:

props.put("auto.offset.reset", "latest"); 
// or props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "latest");

or force it to consume from latest offset using

consumer.seekToEnd();

---