Storm Kafka Spout有很高的滞后
[英]Storm Kafka Spout has very high lag
问题描述
I am building a Spring Application in Java 9 with Apache Storm 1.1.2 and Kafka 0.11 
我正在使用Apache Storm 1.1.2和Kafka 0.11在Java 9中构建Spring应用程序
I've noticed that under a high load (2500 messages per second) that the kafka spout has a very high lag. 我注意到，在高负载（每秒2500条消息）下，kafka喷口的滞后性非常高。 The Kafka Spout has a parallelism hint of 3. The lag is nearly equal to the Spout Committed Offset. Kafka出口的并行度提示为3。滞后时间几乎等于“出口承诺的偏移量”。
This lag sets a cap on the maximum amount of messages the topology can ingest per second which is not great. 此滞后设置了拓扑每秒可以吸收的最大消息量的上限，但这不是很大。 Does anyone know a solution for this? 有谁知道解决方案吗？
UPDATE: I've also noticed that even with 10 workers and a parallelism hint of 4, the load is not divided between different kafka spout executors from what I can tell in the UI. 更新：我还注意到，即使有10个工作程序，并行提示为4，根据我在UI中的判断，负载也不会在不同的kafka spout执行程序之间分配。 The topic has 8 partitions 主题有8个分区
1 个解决方案
解决方案1
0 已采纳 2018-10-04 14:18:16
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.