Apache Camel with RabbitMQ: messages in temp reply queue not being acked when autoAck=false on endpoint configuration

Question

There is an issue I've noticed when configuring InOut capable routes in Camel with the camel-rabbitmq extension. When I set the main queue configuration to autoAck=false the same configuration is then replicated also for the temporary reply queue (it even uses the same prefetch(5) settings, easy to see in the RabbitMQ console). This causes the messages in the temp queue to sit there indefinitely until a server restart.

Virtual host Name                         Features  State Ready Unacked Total incoming deliver / get ack
/test      amq.gen-Hkdx9mckIfMc6JhDI6d-JA AD Excl   idle    2   5   7   0.00/s 0.00/s   0.00/s
/test      amq.gen-eUU7BRI3Ooo4F8Me7HrPnA AD Excl   idle    2   5   7   0.00/s 0.00/s   0.00/s

Even though in the logs I can clearly see that the reply message is received just that the ack doesn't appear to be sent to RabbitMQ to clear our the message from the temp queue. And I've checked in the console that both temp queues have consumers on them so I would have expected Camel to send the ack.

o.a.c.c.r.RabbitMQMessagePublisher  - Sending message to exchange: emailfeedbackExchange with CorrelationId = Camel-ID-VMS-1534332570964-0-11
o.a.c.c.r.r.ReplyManagerSupport  - Received reply message with correlationID [Camel-ID-VMS-1534332570964-0-11]

The question is, how can I prevent this scenario while still keeping my autoAck=false and InOut capable route? Probably I should mention here that there are no errors or the like, the flow works as expected and email processing works perfectly, the only issue is the stale messages on the temp queue.

Our Camel version is 2.20.2 This is the relevant Gradle config for all the Camel components we have:

compile ("org.apache.camel:camel-spring-boot-starter:${camelVersion}")
compile ("org.apache.camel:camel-rabbitmq:${camelVersion}")
compile ("org.apache.camel:camel-amqp:${camelVersion}")

The queue and route configurations:

restentrypointroute:
    restEndpoint: /app
    postEndpoint: /email
    outputEmailEndpoint: rabbitmq://vms:5672/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&bridgeEndpoint=true&concurrentConsumers=3&threadPoolSize=3&channelPoolMaxSize=3&prefetchCount=5&prefetchEnabled=true

emailroutebuilder:
    serviceName: emailroutebuilder
    inputEndpoint: rabbitmq://vms:5672/emailExchange?connectionFactory=rabbitConnectionFactory&autoDelete=false&queue=emailrouteQueue&exchangeType=topic&autoAck=false&bridgeEndpoint=true&concurrentConsumers=3&threadPoolSize=3&channelPoolMaxSize=3&prefetchCount=5&prefetchEnabled=true
    emailProcessor: bean:emailProcessor
    maximumRedeliveries: 5
    redeliveryDelay: 30000

Here is the relevant bit from the RestRouteBuilder implementation:

@Override
public void configure() throws Exception {

restConfiguration().component("restlet").bindingMode(RestBindingMode.json);

    rest(restEndpoint).post(postEndpoint)
      .type(MyRequest.class)
      .route()
      .startupOrder(Integer.MAX_VALUE - 2)
      .process(this::process)
      .choice()
      .when(header(DELIVERYSTATUS_HEADER)
          .isEqualTo(Status.GENERATED)).to(outputEmailEndpoint)
      .when(header(DELIVERYSTATUS_HEADER)
          .isEqualTo(Status.COMPLETED)).to(outputEmailEndpoint, outputArchiveEndpoint).end()
      .endRest();

The process() method adds the DELIVERYSTATUS_HEADER header to the Camel exchange and validates the payload.

The EmailRouteBuilder looks like this:

public void configure() throws Exception {
    super.configure();

    from("direct:" + getServiceName())
            .to(emailProcessor)
            .process(ex -> {
                ex.setOut(ex.getIn());

            });

}

Where the super.configure() call configures exception handling and dead-lettering, startup order, retry counts, max re-deliveries etc. It's quite a bit of code there but if you think something in there might be the cause of this issue I'll post it up. Also, if you need me to add any other configuration please let me know.

From the above it's kind of clear why we need an InOut route with autoAck=false as losing emails is bad from a business standpoint and the REST client would need a response based on how the EmailProcessor got on. Just how to get rid of the stale messages in the temp queues?

EDIT Actually the route only works fine until the prefetch count is exhausted, after that it starts throwing exceptions and the REST client is getting HTTP 500 responses back.

org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis due reply message with correlationID: Camel-ID-VMSYS119-1534407032085-0-284 not received on destination: amq.gen-eUU7BRI3Ooo4F8Me7HrPnA.

Answer 1

As per the comments, this turned out to be a bug in the camel-rabbitmq component and a fix has been now applied to the master branch.

Jira ticket here: https://issues.apache.org/jira/browse/CAMEL-12746

The fix will be available in versions 2.21.3, 2.22.1, 2.23.0 and above.

Edit:

Including the code change in the answer.

TemporaryQueueReplyManager line 139 - always start the consumer of temprary queues with auto acknowledge mode of true .

Changing this:

private void start() throws IOException {         
    tag = channel.basicConsume(getReplyTo(), endpoint.isAutoAck(), this);     
}

To this:

private void start() throws IOException {
     tag = channel.basicConsume(getReplyTo(), true, this);
 }