[英]Spring Integration JMS Consumers not consuming all messages
I have Spring Boot application called the Dispatcher .我有一个名为Dispatcher 的Spring Boot 应用程序。 It runs on 1 Machine and has an embedded ActiveMQ Broker:
它在 1 台机器上运行并具有嵌入式 ActiveMQ 代理:
@Bean
public BrokerService broker(ActiveMQProperties properties) throws Exception {
BrokerService broker = new BrokerService();
broker.setPersistent(false);
broker.addConnector(properties.getBrokerUrl());
return broker;
}
which writes tasks to a JMS queue:将任务写入 JMS 队列:
@Bean
public IntegrationFlow outboundFlow(ActiveMQConnectionFactory connectionFactory) {
return IntegrationFlows
.from(taskQueue())
.bridge(Bridges.blockingPoller(outboundTaskScheduler()))
.transform(outboundTransformer)
.handle(Jms.outboundAdapter(connectionFactory)
.extractPayload(false)
.destination(JmsQueueNames.STANDARD_TASKS))
.get();
}
@Bean
public QueueChannel standardTaskQueue() {
return MessageChannels.priority()
.comparator(TASK_PRIO_COMPARATOR)
.get();
}
// 2 more queues with different names but same config
The Worker Application runs on 10 Machines with 20 cores each and is configured like this: Worker应用程序在 10 台机器上运行,每台机器有 20 个内核,配置如下:
@Bean
public IntegrationFlow standardTaskInbound(ConnectionFactory connectionFactory) {
int maxWorkers = 20;
return IntegrationFlows
.from(Jms.channel(connectionFactory)
.sessionTransacted(true)
.concurrentConsumers(maxWorkers)
.taskExecutor(
Executors.newFixedThreadPool(maxWorkers, new CustomizableThreadFactory("standard-"))
)
.destination(JmsQueueNames.STANDARD_TASKS))
.channel(ChannelNames.TASKS_INBOUND)
.get();
}
// 2 more inbound queues with different names but same config
This is repeated for a 2nd queue, plus 1 special case.这对第二个队列重复,加上 1 个特殊情况。 So there is a total of 401 consumers .
所以总共有401个消费者。
Using JConsole, I can see that there are tasks in the ActiveMQ queue:使用JConsole,可以看到ActiveMQ队列中有任务:
[TODO insert screenshot] 【TODO 插入截图】
As expected, on any Worker machine, there are 20 consumer threads:正如预期的那样,在任何 Worker 机器上,都有 20 个消费者线程:
[TODO insert screenshot] 【TODO 插入截图】
But most if not all of them are idle even though there are still messages in the queue.但是,即使队列中仍有消息,大多数(如果不是全部)也是空闲的。 Using our monitoring tool, I see that about 50 to 400 tasks are being processed at any given time, when the expectation is a constant 400.
使用我们的监控工具,我看到在任何给定时间大约有 50 到 400 个任务正在处理,而期望值为 400。
I also observed that Spring creates AbstractPollingMessageListenerContainer
for each consumer, which seem to result in 1 JMS connection being opened per application per queue per second (33 connections per second).我还观察到 Spring 为每个使用者创建
AbstractPollingMessageListenerContainer
,这似乎导致每个应用程序每秒每个队列打开 1 个 JMS 连接(每秒 33 个连接)。
So I found I do not receive messages in my second consumer which hints at prefetch
being the culprit.所以我发现我没有在我的第二个消费者中收到消息,这暗示
prefetch
是罪魁祸首。 This sounded plausible, so I configured tcp://dispatcher:61616?jms.prefetchPolicy.queuePrefetch=1
on each worker.这听起来似乎有道理,所以我在每个 worker 上配置了
tcp://dispatcher:61616?jms.prefetchPolicy.queuePrefetch=1
。 Then, however, only about 25 tasks were being processed at any point which made no sense to me at all.然而,在任何时候都只有大约 25 个任务在处理,这对我来说根本没有意义。
I don't seem to understand what's going on and since I'm running out of time to investigate, I was hoping that anyone could point me in the right direction.我似乎不明白发生了什么,因为我没有时间去调查了,我希望有人能指出我正确的方向。 Which factors could be the reason?
哪些因素可能是原因? The number of consumers/connections?
消费者/连接的数量? The prefetch?
预取? Anything else?
还要别的吗?
Turned out to be actually caused by the prefetch policy.原来是预取策略导致的。 The correct configuration in my case was to use
tcp://dispatcher:61616?jms.prefetchPolicy.all=0
在我的情况下,正确的配置是使用
tcp://dispatcher:61616?jms.prefetchPolicy.all=0
In my earlier (failed) test I used jms.prefetchPolicy.queuePrefetch=1
but in hindsight I'm not sure whether I configured it at the correct place.在我之前(失败的)测试中,我使用了
jms.prefetchPolicy.queuePrefetch=1
但事后我不确定我是否在正确的位置配置了它。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.