I have a consumer that polls multiple topics. For this question, I've limited one partition per topic. Let's say by the time the consumer started polling, each topic has some data. What is the order of reads?
Is it round-robin? Is it read all from first before the next? I use consumer.poll(N)
to poll.
There is no ordering, as the underlying protocol allows sending requests for multiple partitions in one request.
When you invoke consumer.poll(N)
the client really sends FetchRequest
objects to brokers that are hosting partition leaders (see org.apache.kafka.clients.consumer.internals.Fetcher.createFetchRequests()
) - and it's only one request per node, not per partition.
What is important is that the client can send one FetchRequest for multiple partitions (see protocol spec ).
The ordering is rather complicated. Here is how it works for Kafka 2.6:
Consumer.poll(N)
it returns all the enqueued messages, but at most max.poll.records
(see below)fetch.max.bytes
(or at least one message if available)CompletedFetches
, where one CompletedFetch
contains exactly all the messages of one topic partition from the bufferCompletedFetches
are enqueued (they may contain 0 message or 1000 or more). There will be one CompletedFetch
for every requested topic partitionCompletedFetches
/topic partitions may be mixed up in the final result as opposed to the original assignment orderCompletedFetches
are logically flattened into one big queueConsumer.poll(N)
will read and dequeue at most max.poll.records
from that flattened big queue poll
, another fetch request to all nodes is started, but this time, all the topic partitions that are already in the flattened queue are excludedpoll
callsIn practice that means that you'll have no starving, but you may have a large number of messages from one topic, before you'll get a large number of messages for the next topic.
In tests with a message size of 10 bytes, there were around 58000 messages read from one topic, before roughly the same amount was read from the next. All topics were prefilled with 1 million messages.
Therefore you'll have a kind of batched round robin.
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.