ActiveMQ Pending messages

Question

I have a problem with ActiveMQ similar to this one: http://activemq.2283324.n4.nabble.com/Messages-stuck-in-pending-td4617979.html and already tried the solution posted here.

Some messages seem to get stuck on the queue and can sit there for literally days without being consumed. I have more than enough consumers that are free most of the time, so it's not an issue of "saturation" of consumers.

Upon restart of the ActiveMQ SOME of the pending messages are consumed right away. Just a moment ago I had situation where I had 25 free consumers avaiable for queue (they are visible in the admin panel) with 7 of those "stuck" messages. Four of them were consumed right away but other 3 are still stuck. The other strange thing is - new messages kept coming to queue and were consumed right away, while the 3 old ones were still stuck.

On the consumer side my config in spring looks as follows:

<jms:listener-container concurrency="${activemq.concurrent.consumers}" prefetch="1">
    <jms:listener destination="queue.request" response-destination="queue.response" ref="requestConsumer" method="onRequest"/>
</jms:listener-container>

<bean id="prefetchPolicy" class="org.apache.activemq.ActiveMQPrefetchPolicy">
        <property name="queuePrefetch" value="1" />
</bean>

<bean id="connectionFactory" class="org.apache.activemq.spring.ActiveMQConnectionFactory">
    <property name="brokerURL" value="${activemq.broker.url}?initialReconnectDelay=100&amp;maxReconnectDelay=10000&amp;startupMaxReconnectAttempts=3"/>
    <property name="prefetchPolicy" ref="prefetchPolicy"/>
</bean>

Answer 1

The "stuck" messages are probably considered as "in delivery", restarting the broker will close the connections and, as the message are yet not acknowledged, the broker considers them as not delivered and will deliver them again.

There may be several problem leading to such a situation, most common ones are a problem in transaction / acknowledgment configuration, bad error / acknowledgment management on consumer side (the message is consumed but never acknowledged) or consumer being stuck on an endless operation (for example a blocking call to a third party resource which doesn't respond and there is no timeout handling).