How to prevent kafka to mqtt Apache Camel route from failures?

Question

I have a system that distributes Kafka stream data to MQTT topics. The Apache Camel routes looks like that:

<?xml version="1.0" encoding="UTF-8"?>
<routes xmlns="http://camel.apache.org/schema/spring">
   <route id="KafkaToMQTT">
      <from uri="kafka://mqtt?brokers=localhost:9092" />
      <to uri="micrometer:timer:camel.proxy.kafka.mqtt.stream?action=start" />
      <toD uri="mqtt:mqtt?host=tcp://localhost:1883&amp;publishTopicName=${header.kafka.KEY}" />
      <to uri="log://camel.proxy?groupInterval=100&amp;level=INFO" />
      <to uri="micrometer:timer:camel.proxy.kafka.mqtt.stream?action=stop" />
   </route>
</routes>

And since the data steam starts, for 7 topics of 10 everything is ok, but for messages that should fall into 3 other topics (different between the different runs, but the same at the same camel route run) I'm getting the error message with the stacktrace like below:

19:19:34.195 [Camel (camel-1) thread #4 - KafkaConsumer[mqtt]] ERROR o.a.c.processor.DefaultErrorHandler - Failed delivery for (MessageId: ID-camel-proxy-586888f9b6-dfnwp-1652296571794-0-14406 on ExchangeId: ID-camel-proxy-586888f9b6-dfnwp-1652296571794-0-14404). Exhausted after delivery attempt: 1 caught: java.lang.IllegalStateException: Already connected

Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId              ProcessorId          Processor                                                                        Elapsed (ms)
[KafkaToMQTT       ] [KafkaToMQTT       ] [kafka://mqtt?brokers=kafka%3A9092&groupId=mqtt                                ] [         0]
[KafkaToMQTT       ] [to85              ] [micrometer:timer:camel.proxy.kafka.mqtt.stream?action=start                   ] [         0]
[KafkaToMQTT       ] [toD1              ] [                                                                              ] [         0]

Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.lang.IllegalStateException: Already connected
        at org.fusesource.mqtt.client.CallbackConnection.connect(CallbackConnection.java:135)
        at org.apache.camel.component.mqtt.MQTTEndpoint.connect(MQTTEndpoint.java:315)
        at org.apache.camel.component.mqtt.MQTTProducer.ensureConnected(MQTTProducer.java:108)
        at org.apache.camel.component.mqtt.MQTTProducer.process(MQTTProducer.java:39)
        at org.apache.camel.processor.SendDynamicProcessor$1.doInAsyncProducer(SendDynamicProcessor.java:178)
        at org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:445)
        at org.apache.camel.processor.SendDynamicProcessor.process(SendDynamicProcessor.java:160)
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
        at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
        at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.doRun(KafkaConsumer.java:326)
        at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.run(KafkaConsumer.java:215)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)

For me it is unclear what kind of interaction is inerrupted and if it is just the resource bottleneck like the insufficient connections pool or what else, what can be configured to increase the resources?

Answer 1

It looks like this is <toD/> -related issue because of dynamic nature. Doing the explicit switch with pre-defined value check makes everything working:

<routes
    xmlns="http://camel.apache.org/schema/spring">
    <route id="KafkaToMQTT">
        <from uri="kafka://mqtt?brokers=kafka:9092"/>
        <to uri="micrometer:timer:camel.proxy.kafka.mqtt.stream?action=start"/>
                <when>
                    <simple>${header.kafka.KEY} == "ExpectedValue1"</simple>
                    <to uri="mqtt:mqtt?host=tcp://mqtt:1883&amp;publishTopicName=ExpectedValue1"/>
                </when> 
                <when>
                    <simple>${header.kafka.KEY} == "ExpectedValue2"</simple>
                    <to uri="mqtt:mqtt?host=tcp://mqtt:1883&amp;publishTopicName=ExpectedValue2"/>
                </when> 
        <to uri="log://camel.proxy?groupInterval=100&amp;level=INFO"/>
        <to uri="micrometer:timer:camel.proxy.kafka.mqtt.stream?action=stop"/>      
</route>
</routes>

That isn't so universal, but worked for my goal.s