How can i stop my SimpleMessageListenerContainer from getting stuck in shutdown/restart loop?

Question

I have a SimpleMessageListenerContainer which I am using with RabbitMQ and Java. In most instances I am having no problems, however on some occassions when a message is sent to a queue there appears to be an exception which causes the SMLC to get in a loop trying to shutdown and then restart the queue.

[10/03/15 17:09:38:161 UTC] 00000246 SimpleMessage W org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer run Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: org.springframework.amqp.AmqpIOException: java.io.IOException

[10/03/15 17:09:38:189 UTC] 00000246 SimpleMessage I org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer run Restarting Consumer: tag=[null], channel=Cached Rabbit Channel: AMQChannel(amqp://epa_devint1@xx.xx.xx.xx:5782/,1), acknowledgeMode=AUTO local queue size=0

[10/03/15 17:09:39:164 UTC] 00000256 SimpleMessage W org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer$AsyncMessageProcessingConsumer run Consumer raised exception, processing can restart if the connection factory supports it. Exception summary: com.rabbitmq.client.ShutdownSignalException: connection error; reason: {#method(reply-code=541, reply-text=INTERNAL_ERROR, class-id=0, method-id=0), null, ""}

When i remove the message from the queue through the admin interface then there are no more exceptions.

I believe the cause of the exception is something like a missing header property.

What is the proper way to handle this exception such that the message is removed from the queue and the shutdown/restart logic is exited?

   public SimpleMessageListenerContainer createMessageListenerContainer(Object consumer, String exchangeName, String queueName, String routingKey) {
    TopicExchange exchange = new TopicExchange(exchangeName);
    Queue queue = new Queue(queueName,
            MessagingConstants.RABBIT_MQ_QUEUE_DURABLE,
            MessagingConstants.RABBIT_MQ_QUEUE_EXCLUSIVE,
            MessagingConstants.RABBIT_MQ_QUEUE_AUTO_DELETE,
            MessagingConstants.RABBIT_MQ_QUEUE_ARGUMENTS);

    amqpAdmin.declareExchange(exchange);
    amqpAdmin.declareQueue(queue);
    amqpAdmin.declareBinding(BindingBuilder.bind(queue).to(exchange).with(routingKey));

    SimpleMessageListenerContainer container = new SimpleMessageListenerContainer(connectionFactory);
    container.setQueues(queue);
    container.setConcurrentConsumers(MessagingConstants.RABBIT_MQ_CONCURRENT_CONSUMERS);
    container.setErrorHandler(errorHandler);
    container.setMessageListener(new MessageListenerAdapter(consumer, null));
    container.setAcknowledgeMode(AcknowledgeMode.AUTO);
    container.setAdviceChain(retryAdviceChainFactory.createRequestRequeueExceptionAwareRetryChain(MessagingConstants.RABBIT_MQ_RETRY_ATTEMPTS));
    container.setChannelTransacted(true);
    container.setTaskExecutor(taskExecutor);
    container.setTransactionManager(transactionManager);
    return container;
}

@Override
public void afterPropertiesSet() {
    com.rabbitmq.client.ConnectionFactory rabbitFactory = new com.rabbitmq.client.ConnectionFactory() {
        protected void configureSocket(Socket socket) throws IOException {
            super.configureSocket(socket);
            socket.setSoTimeout(propertiesHolder.getRabbitMQSocketTimeoutMS());
        }
    };
    rabbitFactory.setConnectionTimeout(propertiesHolder.getRabbitMQConnectionTimeoutMS());
    rabbitFactory.setRequestedHeartbeat(propertiesHolder.getRabbitMQRequestedHeartbeatInSeconds());

    CachingConnectionFactory cachingFactory = new CachingConnectionFactory(rabbitFactory);
    cachingFactory.setAddresses(propertiesHolder.getRabbitMQHost());
    cachingFactory.setPort(propertiesHolder.getRabbitMQPort());
    cachingFactory.setUsername(propertiesHolder.getRabbitMQUsername());
    cachingFactory.setPassword(propertiesHolder.getRabbitMQPassword());
    cachingFactory.setChannelCacheSize(propertiesHolder.getRabbitMQChannelCacheSize());

    connectionFactory = cachingFactory;
    retryAdviceChainFactory = new RetryAdviceChainFactory();
    amqpAdmin = new RabbitAdmin(connectionFactory);

    errorHandler = new ErrorHandler() {
        @Override
        public void handleError(Throwable e) {
            LOG.error("Error occurred", e);
        }
    };

    TopicExchange deadLetterExchange = new TopicExchange(MessagingConstants.RABBIT_MQ_DEAD_LETTER_EXCHANGE);
    Queue deadLetterQueue = new Queue(
            MessagingConstants.RABBIT_MQ_DEAD_LETTER_QUEUE,
            MessagingConstants.RABBIT_MQ_QUEUE_DURABLE,
            MessagingConstants.RABBIT_MQ_QUEUE_EXCLUSIVE,
            MessagingConstants.RABBIT_MQ_QUEUE_AUTO_DELETE,
            MessagingConstants.RABBIT_MQ_QUEUE_ARGUMENTS);
    amqpAdmin.declareExchange(deadLetterExchange);
    amqpAdmin.declareQueue(deadLetterQueue);
    amqpAdmin.declareBinding(BindingBuilder.bind(deadLetterQueue).to(deadLetterExchange).with("#"));
    messageRecoverer = new DeadLetterMessageRecoverer(rabbitTemplate(), deadLetterExchange);
}

JDK: 1.6 spring-rabbit: 1.1.3.Release spring-framework: 3.2.1.Release

Answer 1

Your problem is caused by the transactional nature of your message handling. I mean this:

container.setTransactionManager(transactionManager);

What is happening is, that in some cases (probably some issue with message header, as you suggested) you encounter some error, and that is not being handled properly. The error propagates through and reaches the txManager, which in turn:

• does not acknowledge the message, and returns the message to the queue
• considers your current connection corrupted , and closes it, so it can open a brand new one

Now this behavior above absolutely makes sense when you had an issue which is not a direct consequence of your message. Let's say you encountered a timeout while processing the message; the application was shutting down while processing it; in all these cases redelivering the message later OR to another node does make sense.

In your case however does not matter when and does not matter which node receives the faulty message, you will get the same problem, and you need manually delete that message. In order to avoid this situation you can:

• programmatically filter out invalid messages and do not return them to the queue either by

  • doing checks before processing it (prevent the fatal error happening)
  • handling the fatal errors like this in a different way (proper error handling)

• use RabbitMQ features to deal with errors depending on your requirements

  • Dead Letter Queue
  • Broker side validation
  • Probably TTL would work too