Kafka不会消耗所有生成的数据

Question

I have a single instance of kafka installed on a VM with 8 cores and 32GB RAM.

I write to it (produce) from 10 different machines and consume from one machine, all of which are in the same network.

The size of the data that I produce is ~ 35MBit / s.

For some reason, most of the time I can't consume more than ~ 10MBit /s (for limited periods of time I do manage to consume all the produced data), even though the kafka AND the consumer servers are mostly idle (therefore I don't think it's a retention problem).

Could kafka be ignoring some of the produced data?

Some parameter values that might be useful for analysis:

num.network.threads=32
num.io.threads=16
message.max.bytes=2147483647
num.partitions=10
log.retention.ms=120000 (2 minutes)

Answer 1

Your retention time is way too low. If your consumer ever falls further than 2 minutes behind any one of the 10 producers then the messages will be lost. Try 24 hours or at least as much as you have disk space to fill. The default retention period is 7 days. Keeping messages for a longer period will also help you debug if they are all getting into the topic successfully.