Storm Kafka Spout has very high lag

Question

I am building a Spring Application in Java 9 with Apache Storm 1.1.2 and Kafka 0.11

I've noticed that under a high load (2500 messages per second) that the kafka spout has a very high lag. The Kafka Spout has a parallelism hint of 3. The lag is nearly equal to the Spout Committed Offset.

This lag sets a cap on the maximum amount of messages the topology can ingest per second which is not great. Does anyone know a solution for this?

UPDATE: I've also noticed that even with 10 workers and a parallelism hint of 4, the load is not divided between different kafka spout executors from what I can tell in the UI. The topic has 8 partitions

Answer 1

请参阅Apache Storm Kafka喷口滞后问题