如何仅获取php kafka中的最新记录

Question

I am using arnaud-lb/php-rdkafka as PHP kafka client. I am using Windows 10 and php 7.4. I am trying to get only the latest records via consumer, but it is returning all records. I am not sure whether it is not committing offset as per my following code or there is any other reason. I was also trying to find how to commit offset if it is not automatically happening but could not figure out anything.

$conf = new RdKafka\Conf();
//myConsumerGroup
$conf->set('group.id', 'myConsumerGroup'.date('Ymdhis'));
$rk = new RdKafka\Consumer($conf);
$rk->addBrokers("127.0.0.1");
$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 100);
$topicConf->set('offset.store.method', 'broker');
$topicConf->set('auto.offset.reset', 'earliest');
$topic = $rk->newTopic("TestTopic", $topicConf);
//RD_KAFKA_OFFSET_BEGINNING RD_KAFKA_OFFSET_STORED
$topic->consumeStart(0, RD_KAFKA_OFFSET_STORED);
echo '<pre>';
while (true) {
    $message = $topic->consume(0, 2*10000);
    switch ($message->err) {
        case RD_KAFKA_RESP_ERR_NO_ERROR:
            print_r($message);
            break;
        case RD_KAFKA_RESP_ERR__PARTITION_EOF:
            echo "No more messages; will wait for more\n";
            break;
        case RD_KAFKA_RESP_ERR__TIMED_OUT:
            echo "Timed out\n";
            break;
        default:
            throw new \Exception($message->errstr(), $message->err);
            break;
    }
}

1. If I give group id = myConsumerGroup then it does not return anything.
2. If I give RD_KAFKA_OFFSET_BEGINNING instead of RD_KAFKA_OFFSET_STORED then it returns all records from start.
3. In the above sample code, in both RD_KAFKA_OFFSET_BEGINNING and RD_KAFKA_OFFSET_STORED, it is returning all records.
4. If I use RD_KAFKA_OFFSET_BEGINNING and group id "myConsumerGroup" then it returns all records from the beginning.

Answer 1

I was also trying to find how to commit offset

Set $conf->set('enable.auto.commit', 'true'); or use an explicit $rk->commit($message); RdKafka\KafkaConsumer::commit .