camel jms consumer throws RejectedExecutionException, but only on one particular queue

Question

We have an activemq broker that we use for exchanging messages between backend services and frontend devices. It's effectively two brokers in one, glued together by camel.

For the backend there's a JMS broker, for the frontend there's an mqtt broker. We use camel to establish routes that cross the line, usually from a JMS queue to an mqtt topic, or the other way around. We've had this service running for years now. Every now and then a feature would require a new routing, so we put it in. That all went well, until now.

Now, for some reason, one of the new jms queues we've added is not forwarding its messages to the mqtt topic, rather it raises an error:

[Camel (camel-1) thread #10 - JmsConsumer[bmetry-command]] o.a.camel.processor.DefaultErrorHandler : Failed delivery for (MessageId: ID:ip-172-31-26-161.eu-west-1.compute.internal-42307-1669731056848-1:115:5:1:406 on ExchangeId: ID-broker-yellow-webcam-1669798394280-0-5852). Exhausted after delivery attempt: 1 caught: java.util.concurrent.RejectedExecutionException

java.util.concurrent.RejectedExecutionException: null
at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:144) ~[camel-jms-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:178) ~[camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445) ~[camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:160) ~[camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) ~[camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) [camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) [camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) [camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) [camel-core-2.22.0.jar!/:2.22.0]
at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:113) [camel-jms-2.22.0.jar!/:2.22.0]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:736) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:696) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:674) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:318) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:257) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1189) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1179) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:1076) [spring-jms-5.2.8.RELEASE.jar!/:5.2.8.RELEASE]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_292]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_292]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_292]

I can't really get too much useful information out of that stack trace, other than the queue it fails for.

There's a couple of weird things here:

• As mentioned, this is the only JMS queue on which that happens, and there's a couple dozen of them.
• On this queue however, it happens consistently . There's not a single message going through, while all the others are working fine.
• When connecting with an mqtt client and publishing to the topic the jms queue is supposed to deliver to, the message arrives without issue, so it's clearly the JMS end that's the problem.
• This particular queue doesn't have much to do most of the time, but for short periods, it can have one of the highest throughputs in the broker (though that doesn't mean much. If it gets more than 1 message a second, it's really busy, and the messages are very short). However, experimentation has shown that it won't be working any better with less throughput (ie it still doesn't work at all).

The real bummer though is that an identical broker in the development environment works perfectly fine, even with the same or higher throughput on that queue. It has a lot less load overall, obviously, but the fact that everything works fine except for this one specific queue kind of doesn't sound like a load issue.

Let's get to the code, which is very unspectacular. The whole thing is a spring-boot application with camel inside. The two brokers are launched and configured like this:

// JMS broker component:
@Component
class JmsBroker(jmsProperties: JmsProperties) {

    private val broker = BrokerService().apply {
        isPersistent = false
        isAdvisorySupport = false
        addConnector("nio://${jmsProperties.address}:${jmsProperties.port}")
        brokerName = "jms"
    }

    @PostConstruct
    fun start() {
        broker.start()
    }

    @PreDestroy
    fun stop() {
        broker.stop()
    }
}

// MQTT config and bean:
@Configuration
class MqttConfig {
    val log = logger()

    @Bean
    fun sslConfig(mqttProperties: MqttProperties): SpringSslContext? {
        return if (isTlsActive(mqttProperties)) {
            SpringSslContext().apply {
                keyStore = "./${mqttProperties.keystore}"
                keyStorePassword = mqttProperties.keystorePassword
            }
        } else null
    }

    @Bean
    fun mqttBroker(mqttProperties: MqttProperties,
                   authenticationService: AuthenticationService,
                   ssl: SslContext?): BrokerService {
        return BrokerService().apply {

            isPersistent = false
            isAdvisorySupport = false
            brokerName = "mqtt"
            plugins = listOf(WebcamServiceAuthenticationPlugin(authenticationService, mqttProperties.internalPort)).toTypedArray()
            addConnector("nio://127.0.0.1:${mqttProperties.internalPort}")

            if (isTlsActive(mqttProperties)) {
                log.info("Starting MQTT connector using TLS")
                sslContext = ssl!!
            } else {
                log.info("Starting a plain text MQTT connector")
            }
            addConnector(makeHostUri(mqttProperties))
        }
    }

    private fun makeHostUri(mqttProperties: MqttProperties): String =
            (if (isTlsActive(mqttProperties)) "mqtt+nio+ssl://" else "mqtt+nio://") +
                    "${mqttProperties.address}:${mqttProperties.port}"


    private fun isTlsActive(mqttProperties: MqttProperties) =
            mqttProperties.keystore.isNotEmpty()

}

And here's the whole camel setup and configuring the routes. I have only put in the route that is not working here, so you can have a look at it, but it's essentially identical to a dozen others in the same config:

@Component
class BrokerRouteBuilder(camelContext: CamelContext,
                         mqttProperties: MqttProperties,
                         jmsProperties: JmsProperties,
                         private val authenticationService: AuthenticationService,
                         private val objectMapper: ObjectMapper)
    : RouteBuilder() {

    private val jms = "jmsbroker"
    private val mqtt = "mqttbroker"

    init {
        camelContext.addComponent(jms, activeMQComponent().apply {
            setConnectionFactory(PooledConnectionFactory("nio://127.0.0.1:${jmsProperties.port}")
                    .apply { maxConnections = jmsProperties.maxInternalConnections })
            setCacheLevelName("CACHE_CONSUMER")
        })
        camelContext.addComponent(mqtt, activeMQComponent().apply {
            setConnectionFactory(PooledConnectionFactory("nio://127.0.0.1:${mqttProperties.internalPort}")
                    .apply { maxConnections = mqttProperties.maxInternalConnections })
            setCacheLevelName("CACHE_CONSUMER")
        })
        objectMapper.registerModule(JodaModule())
    }

    override fun configure() {

        // takes message from the queue and directs it to an mqtt topic for the user noted 
        // in the header of the jms message
        from("$jms:queue:bmetry-command")
                    .id("bmetry-command")
                    .log("Sending bmetry command to user '\${header.user}'")
                    .toD("$mqtt:topic:\${header.user}.bmetry-command")

}

And that's pretty much all I got. I would appreciate any help that could point me in the right direction to find a solution for this. I've googled, obviously, but can't find the problem in the right context, and am too unfamiliar with camel internals to really dig down on this on my own.

Additional information

I've dug out the line in the camel source code that the stack trace refers to, and it all seems to come down to ServiceSupport.isRunAllowed() always returning false on that jms producer. Here's the function in question:

    public boolean isRunAllowed() {
        // if we have not yet initialized, then all options is false
        boolean unused1 = !started.get() && !starting.get() && !stopping.get() && !stopped.get();
        boolean unused2 = !suspending.get() && !suspended.get() && !shutdown.get() && !shuttingdown.get();
        if (unused1 && unused2) {
            return false;
        }
        return !isStoppingOrStopped();
    }

So either the exchange is not initialised, or it is already stopped. I have no idea why either could be the case. There's nothing inherently different about this route when compared to many other routes in the same application. Also, as mentioned, it isn't an issue in the development environment.

I have found the same behaviour described here, but the suggested change didn't solve the problem in this case: https://github.com/camelinaction/camelinaction2/issues/158

Progress, but maybe not

So I've upgraded spring-boot and camel to the latest versions I could afford. That would be camel 3.14, because the thing still has to run on java 8.

This has changed the behaviour slightly. The error now appears in the JmsConsumer, ie earlier, so there's still a potential that I figure this one out and then the old one will just be happening again. It does seem unlikely that the two wouldn't be related, though, considering they're behaving the exact same way apart from where exactly they occur. Always on that one route, and only on that route, and only in the production environment.

The new error I'm seeing is this:

java.lang.NoClassDefFoundError: org/apache/camel/util/MessageHelper
    at org.apache.camel.processor.RedeliveryErrorHandler.logFailedDelivery(RedeliveryErrorHandler.java:1308) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.RedeliveryErrorHandler.deliverToFailureProcessor(RedeliveryErrorHandler.java:1109) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:474) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:138) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.Pipeline.process(Pipeline.java:101) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97) ~[camel-core-2.22.0.jar!/:2.22.0]
    at org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:113) ~[camel-jms-2.22.0.jar!/:2.22.0]

What is very weird here is that the stacktrace is referring to camel-core 2.22.0. THat was the camel version that was running on there before. But I've checked everything, including the jar file that's been deployed and running on the instance, and there's only camel 3.14 jars in the dependencies. It's a bootjar, obviously, and there's no trace of any camel 2.22 stuff in that jar. And yet here it is in the stack trace. Can somebody explain how that happens?

Answer 1

All in all, I can say that the issue was solved by upgrading to spring-boot 2.7 and camel 3.14 (from sb 2.3.3 and camel 2.22).

I got held up by some very weird but most probably unrelated behaviour probably originating somewhere in deployment, but once that was smoothed out, the original issue was gone.