Kafka 1.1 多消费者群体滞后

Question

Our broker is receiving a large number of messages (about 15-30 MB/sec), and we would like to ingest these messages in real time and do some processing. Each message is a few hundreds kB.

Our processing pipelines use pools of threads, so we would like the records being ingested from a single poll to contain a large batch of messages such that our thread pools do not have to deal with processing small batches frequently and run the risk of running out of threads. Currently, we are controlling this by modifying fetch.min.bytes , receive.buffer.bytes , and max.partition.fetch.bytes configurations of our Kafka Consumer.

We currently do not have the capability of having a single consumer which can demux the messages to different pipelines. As such, we assign a single consumer to a pipeline and each consumer is assigned to its own group.

The problem we are having is that once we start ingesting for multiple pipelines, each having its own consumer in its own group, our ingestion rate starts to lag behind the producer. What's interesting is, when we have a single pipeline running, we do not have the lag issue. Our application is for real-time or near real-time analysis, so ultimately, we would like the lag to be 0 or as close to 0 as possible.

What would be the best way of configuring the consumers such that when they run in different group simultaneously, we can minimize the lags as much as possible?

Answer 1

This probably points to a configuration issue. Kafka is designed to blast data down to consumers as fast as possible, assuming the messages being consumed are still in page cache . If they are not in page cache anymore, it means the messages were produced some time ago, and they now only exist on disk logs and not in page cache, then you'll definitely see a slowdown, because Kafka has to go and read the logs from disk, which is thousands of times slower than reading from memory.

If you don't want to deal with small batches, on top of the properties you mention that you have tuned, you should also pay attention to how long your consumer polls are. Anything at 50ms or higher should be plenty. However, I've seen clients using 1ms as the poll interval, which effectively slows down the consuming, since it doesn't allow the consumer enough time to get as much data as it could.

One final recommendation. Do not execute data processing/validation on the same thread that the kafka consumer is consuming on. Sometimes, people do expensive processing on the same consumer thread, and without realizing it, slow down the consumption. That consumer thread should simply grab the messages from Kafka, ideally not even deserialize them, just grab the bytes (or String, or whatever your serialized format is), and dump it on a thread-safe queue where it can be deserialized and processsed off-thread. That will ensure the consuming thread can poll as fast as possible, only limited by the available CPU.

Finally, there's a lot of great recommendations and tips on the official Javadoc for KafkaConsumer here: https://kafka.apache.org/20/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html