[英]ActiveMQ/Camel failover for request/reply pattern using temp-queues in Network of brokers - can not publish to deleted temp-queue
Due to scaling reasons we recently switched from a single ActiveMQ broker to a network of brokers. 由于扩展原因,我们最近从单个ActiveMQ代理切换到代理网络。 While for the most part everything works exactly as intended there is one weird issue we have encountered after a fresh deployment of the brokers during the day:
虽然在大多数情况下一切都按照预期完成,但在白天经纪人重新部署后我们遇到了一个奇怪的问题:
First just for the tech stack we are using ActiveMQ 5.12.1 and Camel 2.13.4 for the integration between the java method and a JMS endpoint. 首先,对于技术堆栈,我们使用ActiveMQ 5.12.1和Camel 2.13.4来实现java方法和JMS端点之间的集成。 The broker side is a network of brokers currently consisting of 3 members using the following configuration
经纪人方是一个经纪人网络,目前由3名成员组成,使用以下配置
<broker useJmx="${activemq.expose.jmx}" persistent="false"
brokerName="${activemq.brokerName}" xmlns="http://activemq.apache.org/schema/core">
<sslContext>
<amq:sslContext keyStore="${activemq.broker.keyStore}"
keyStorePassword="${activemq.broker.keyStorePassword}"
trustStore="${activemq.broker.trustStore}"
trustStorePassword="${activemq.broker.trustStorePassword}" />
</sslContext>
<systemUsage>
<systemUsage>
<memoryUsage>
<memoryUsage limit="${activemq.memoryUsage}" />
</memoryUsage>
<tempUsage>
<tempUsage limit="${activemq.tempUsage}" />
</tempUsage>
</systemUsage>
</systemUsage>
<destinationPolicy>
<policyMap>
<policyEntries>
<policyEntry queue=">" enableAudit="false">
<networkBridgeFilterFactory>
<conditionalNetworkBridgeFilterFactory
replayWhenNoConsumers="true" />
</networkBridgeFilterFactory>
</policyEntry>
</policyEntries>
</policyMap>
</destinationPolicy>
<networkConnectors>
<networkConnector name="queues"
uri="static:(${activemq.otherBrokers})"
networkTTL="2" dynamicOnly="true"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="false">
<excludedDestinations>
<topic physicalName=">" />
</excludedDestinations>
</networkConnector>
<networkConnector name="topics"
uri="static:(${activemq.otherBrokers})"
networkTTL="1" dynamicOnly="true"
decreaseNetworkConsumerPriority="true"
conduitSubscriptions="true">
<excludedDestinations>
<queue physicalName=">" />
</excludedDestinations>
</networkConnector>
</networkConnectors>
<transportConnectors>
<transportConnector
uri="${activemq.protocol}${activemq.host}:${activemq.tcp.port}?needClientAuth=true"
updateClusterClients="true" rebalanceClusterClients="true" />
<transportConnector
uri="${activemq.websocket.protocol}${activemq.websocket.host}:${activemq.websocket.port}?needClientAuth=true"
updateClusterClients="true" rebalanceClusterClients="true" />
</transportConnectors>
</broker>
with the following placeholder values 使用以下占位符值
activemq.tcp.port=9000
activemq.protocol=ssl://
activemq.brokerName=activemq-server1.com
activemq.expose.jmx=true
activemq.otherBrokers=ssl://server2.com:9000,ssl://server3.com:9000
activemq.websocket.port=9001
activemq.websocket.protocol=stomp+ssl://
activemq.websocket.host=server1.com
activemq.memoryUsage=1gb
activemq.tempUsage=1gb
On the client side the following camel configuration is being used 在客户端,正在使用以下camel配置
<bean id="xxx.activemq.redeliveryPolicy" class="org.apache.activemq.RedeliveryPolicy">
<property name="maximumRedeliveries" value="0" />
</bean>
<bean id="xxx.activemq.jmsConnectionFactory" class="org.apache.activemq.ActiveMQSslConnectionFactory">
<property name="trustStore" value="${activemq.broker.trustStore}" />
<property name="trustStorePassword" value="${activemq.broker.trustStorePassword}" />
<property name="keyStore" value="${activemq.broker.keyStore}" />
<property name="keyStorePassword" value="${activemq.broker.keyStorePassword}" />
<property name="brokerURL" value="${activemq.broker.url}" />
<property name="redeliveryPolicy" ref="xxx.activemq.redeliveryPolicy" />
</bean>
<bean id="xxx.activemq.jmsConfiguration" class="org.apache.activemq.camel.component.ActiveMQConfiguration">
<property name="receiveTimeout" value="6000" />
<property name="connectionFactory" ref="xxx.activemq.pooledConnectionFactory" />
</bean>
<bean id="xxx.activemq.pooledConnectionFactory"
class="org.apache.activemq.pool.PooledConnectionFactory"
init-method="start" destroy-method="stop">
<property name="maxConnections" value="8" />
<property name="idleTimeout" value="0" />
<property name="timeBetweenExpirationCheckMillis"
value="10000" />
<property name="connectionFactory"
ref="xxx.activemq.jmsConnectionFactory" />
</bean>
<bean id="xxx.activemq.jms.abstractComponent" abstract="true"
class="org.apache.activemq.camel.component.ActiveMQComponent">
<property name="configuration"
ref="xxx.activemq.jmsConfiguration" />
<property name="connectionFactory"
ref="xxx.activemq.pooledConnectionFactory" />
<property name="allowNullBody" value="true" />
<property name="transferException" value="true" />
<property name="defaultTaskExecutorType"
value="#{T(org.apache.camel.component.jms.DefaultTaskExecutorType).ThreadPool}" />
<property name="requestTimeout" value="5000" />
</bean>
<bean id="xxx.activemq.jms.queue"
parent="xxx.activemq.jms.abstractComponent">
<property name="concurrentConsumers" value="2" />
<property name="maxConcurrentConsumers" value="2" />
</bean>
with a connection url of 有一个连接网址
activemq.broker.url=failover:(ssl://server1.com:9000,ssl://server2.com:9000,ssl://server3.com:9000)?randomize=true
The request/reply EIP is achieved by having the producer set an according jmsReplyTo header and having camel default to the InOut using temp-queues. 请求/回复EIP是通过让生产者设置一个相应的jmsReplyTo头并使用temp-queues将camel默认设置为InOut来实现的。
Before the deployment all messaging was working as intended, however afterwards for some request/reply queues we would start to get timeouts on the producer side. 在部署之前,所有消息都按预期工作,但是之后对于某些请求/回复队列,我们将开始在生产者端获得超时。 The following entries showed up in the logs:
以下条目显示在日志中:
On producer side 在生产者方面
Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 5000 millis due reply message with correlationID: Camel-ID-xxx-intranet-phs-49404-1457684675710-8-11 not received on destination: temp-queue://ID:xxx.intranet.phs-41986-1457684806758-1:3:1.
Exchange[Message: BeanInvocation public abstract xxx.xxx.rapi.dto.RemoteDTO xxx.xxx.xxx.facade.RemoteFacade.findRemoteDTO(java.lang.String,java.lang.Long) with [xxx, 31333]]]
and on consumer side: 在消费者方面:
Caused by: javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:xxx.intranet.phs-41986-1457684806758-1:3:1
We have since then done some research and found that the problem would show up whenever an arbitrary broker of the network is shut down and then only for those producers that had a temp-queue open for a reply when the shutdown hit and they failover to a new broker. 从那时起,我们进行了一些研究,发现每当网络的任意代理关闭时,问题都会出现,然后只有那些在关机命中时有临时队列打开响应并且故障转移到新经纪人。 Afterwards the problem would persist for this producer until he is restarted.
之后问题会持续到这个制作人,直到他重新启动。 Once he joins back after a restart everything is back to normal.
一旦他重新启动后重新加入,一切都恢复正常。 The problem is also described on grokbase as well as on two topics here activemq-failover-with-temporary-queues-on-a-network-of-brokers and activemq-how-to-handle-broker-failovers-while-using-temporary-queues .
这个问题也在grokbase以及两个主题上描述了activemq-failover-with-temporary-queues-on-a-network-of-broker和activemq-how-to-handle-broker-failovers-while-using-临时队列 。 We have tried the one solution given in activemq-how-to-handle-broker-failovers-while-using-temporary-queues to set the cache timeout but did not get any results from it, the other suggested option to turn of advisory listening for clients is not really an option in our setup since we want to make use of features such as clusterRebalancing for easier adding of additional brokers to the network during runtime.
我们尝试了在activemq-how-to-handle-broker-failovers-while-using-temporary-queues中给出的一个解决方案来设置缓存超时但没有从中得到任何结果,另一个建议选择转向咨询监听因为我们希望利用clusterRebalancing等功能在运行时更容易地向网络添加其他代理,因此对于客户端来说,我们的设置并不是真正的选项。
We have also found some JIRA issues on camel and ActiveMQ side like CAMEL-3193 that describe this issue and apparently fix them for versions where ours are newer so we are quite puzzled. 我们还发现了一些关于Camel和ActiveMQ方面的JIRA问题,如CAMEL-3193描述了这个问题,并且显然已将它们修复为我们更新的版本,所以我们非常困惑。 Currently we are considering switching to exclusive reply queues over temporary queues to address this issue, but first wanted to ask if maybe we are just missing some configuration somewhere.
目前我们正在考虑切换到临时队列上的独占回复队列来解决这个问题,但首先想问一下我们是否只是遗漏某些配置。
If you need any additional information please just ask! 如果您需要任何其他信息,请询问!
See AMQ-5469 : Temporary queues get deleted when the connection that established them drops. 请参阅AMQ-5469 :当建立连接的连接断开时,将删除临时队列。 This happens when you shutdown a broker.
关闭代理时会发生这种情况。
The solution can be one of two options: 解决方案可以是以下两种选择之一:
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.