Kafka 消费者没有分配到任何分区

Question

I am calling this class in a service which runs for every half an hour. I just want to get the last message from a list of topic. But randomly I am not getting the latest record. Rather, it shows the offset value as 0 in that case. Also the newly assigned partitions are coming as empty Setting newly assigned partitions [].

 try {
            consumer.subscribe(Collections.singletonList(topic));

            ConsumerRecords last= consumer.poll(Duration.ofSeconds(2).toMillis());

            consumer.assignment().forEach(System.out::println);
            AtomicLong maxTimestamp = new AtomicLong();
            AtomicReference<ConsumerRecord<String, String>> latestRecord = new AtomicReference<>();

            // get the last offsets for each partition
            consumer.endOffsets(consumer.assignment()).forEach((topicPartition, offset) -> {
                logger.info("offset: "+offset);

                // seek to the last offset of each partition
                consumer.seek(topicPartition, (offset==0) ? offset:offset - 1);
                topicoffset=topicoffset+offset;
                // poll to get the last record in each partition
                consumer.poll(Duration.ofSeconds(10).toMillis()).forEach(record -> {
                    
                    // the latest record in the 'topic' is the one with the highest timestamp
                    if (record.timestamp() > maxTimestamp.get()) {
                        maxTimestamp.set(record.timestamp());
                        latestRecord.set(record);
                    }
                });
            });
            
            ConsumerRecord<String, String> lastrecord =latestRecord.get();

The consumer

[Consumer clientId=consumer-1, groupId=get-offset-command-test-1] Discovered coordinator kafka.test.com:9092 (id: 2147483647 rack: null)
 [Consumer clientId=consumer-1, groupId=get-offset-command-test-1] Revoking previously assigned partitions []
 [Consumer clientId=consumer-1, groupId=get-offset-command-test-1] (Re-)joining group
 [Consumer clientId=consumer-1, groupId=get-offset-command-test-1] Successfully joined group with generation 2
[Consumer clientId=consumer-1, groupId=get-offset-command-test-1] Setting newly assigned partitions []

Answer 1

have two instances of the jar containing the above class having same group id... the topics are different but the group id is same... I get empty list only when both the instances are running.If i down one instance,then the other one is working fine.

When you start the "other" instance to subscribe to a different topic, but in the same group, then the entire group will rebalance. This will unassign (and eventually reassign) partitions to the "first instance".

It's unlikely both instances will have an empty list at the same time, but there is a possibility that one instance will grab all partitions from the group, but only for the topic it has been written to subscribe to.

When using the same group across consumer instances, best practice would be to always subscribe to the same topics.

Since there's no clear reason to use the same group to get the "latest" record from any topic, I suggest using a random UUID for the group and then disabling any auto commits in the consumer config