I'm using the kafka-node HighLevelConsumer, and am having problems where I always receive duplicate messages on startup.
In order to maintain processing sequence, my consumer simply appends messages to a work queue, and I process the events serially. I pause the consumer if I hit a queue high-water mark, I have auto-commit disabled, and I commit "manually" after my client code fully processes each event.
Despite committing, on startup, I always get the last (previously committed) message from one or more partitions (depending on how many other HLCs are running in my group). I was a little surprised that the HLC wouldn't give me (committed+1) but I decided to just "ignore" messages that had an offset earlier than the offset committed. As a quick test,
offset.fetchCommits('fnord', [{topic:'test', partition: 0},
{topic:'test', partition: 1},
{topic:'test', partition: 2},
{topic:'test', partition: 3}], ...
This works if my payload list matches the number of partitions defined. If I exceed the number of partitions, I get a [BrokerNotAvailableError: Could not find the leader]
error.
I dug into the kafka-node source, and there's an undocumented call I was able to use to get the partition info:
client.loadMetadataForTopics(['test'], function(err, results) {..}
(I don't love calling something that doesn't appear to be a documented part of the public API, and I'm uncomfortable with the rather raw-feeling mixed array nature of the returned results, but it solves my problem for the moment.)
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.