[英]kafka consumer get a message which was consumed before
我在XD中有一個消費者工作,一旦收到其他生產者工作產生的消息,該工作便會完成。 我每天都會觸發這些工作。 我發現有時該消費者收到一條消息,該消息以前曾被使用過。
記錄如下:
####OK
2019-06-28 02:06:13+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========consumed poll data ConsumerRecord(topic = my_consumer_topic, partition = 0, leaderEpoch = 0, offset = 4, CreateTime = 1561658772877, serialized key size = -1, serialized value size = 30, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = message_from_producer) ==================
2019-06-28 02:06:13+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========message is message_from_producer, task startTime is 1561658700108, timestamp is 1561658772877 ==================
####NG
2019-06-29 17:07:14+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========consumed poll data ConsumerRecord(topic = my_consumer_topic, partition = 0, leaderEpoch = 0, offset = 0, CreateTime = 1561399136840, serialized key size = -1, serialized value size = 30, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = message_from_producer) ==================
2019-06-29 17:07:14+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========message is message_from_producer, task startTime is 1561799100282, timestamp is 1561399136840 ==================
####OK
2019-06-29 22:16:58+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========consumed poll data ConsumerRecord(topic = my_consumer_topic, partition = 0, leaderEpoch = 2, offset = 5, CreateTime = 1561817817702, serialized key size = -1, serialized value size = 30, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = message_from_producer) ==================
2019-06-29 22:16:58+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========message is message_from_producer, task startTime is 1561817528447, timestamp is 1561817817702 ==================
####NG
2019-07-02 02:05:09+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========consumed poll data ConsumerRecord(topic = my_consumer_topic, partition = 0, leaderEpoch = 0, offset = 0, CreateTime = 1561399136840, serialized key size = -1, serialized value size = 30, headers = RecordHeaders(headers = [], isReadOnly = false), key = null, value = message_from_producer) ==================
2019-07-02 02:05:09+0800 INFO inbound.job:Consumer_Job_In_XD-redis:queue-inbound-channel-adapter1 myConsumer.ConsumeTasklet - ==========message is message_from_producer, task startTime is 1562004300372, timestamp is 1561399136840 ==================
看起來它多次獲得了offset = 0消息。
Kakfa版本(1.0.0)
使用者手動提交偏移量。(consumer.commitSync();)
僅設置以下屬性:
bootstrap.servers
auto.offset.reset=earliest
group.id
client.id
Properties config = new Properties();
config.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
config.put("auto.offset.reset", "earliest");
config.put("group.id", group);
config.put("client.id", config.getProperty("group.id") + "_" + System.currentTimeMillis());
config.put("enable.auto.commit", false);
try {
consumer = new KafkaConsumer<>(config);
consumer.subscribe(tList);
while (true) {
ConsumerRecords<?, ?> records = consumer.poll(10000);
for (ConsumerRecord<?, ?> record : records) {
//.........
consumer.commitSync();
}
if (matched)
break;
}
} finally {
consumer.close();
}
在Kafka 1.1中,默認情況下,偏移量僅保留24小時,因為offsets.retention.minutes
設置為1440。
因此,如果您停止使用消費者超過24小時,則在重新啟動時,可能會刪除已提交的偏移量,從而迫使消費者使用auto.offset.reset
來查找新位置。
由於對於許多人來說太短了,從Kafka 2.0開始, offsets.retention.minutes
現在設置為10080(7天)。
您應該更改代理配置,以允許將偏移量保留更長的時間,或者更新到更新的Kafka版本。
嘗試設置auto.offset.reset = latest,以這種方式,在重新啟動后,使用者將在最近的提交偏移量之后開始消耗。
更多信息在這里https://kafka.apache.org/documentation/#consumerconfigs
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.