ActiveMQ - Slow kaha DB access causing ActiveMQ to not respond fast enough

Question

I have a wso2 ESB talking to activeMQ (persistent messaging). Sometimes, the threads on ESB stack up because all the threads are waiting for activeMQ to respond to the various calls being made to it. Eventually the calls error out.

Meanwhile in the ActiveMQ logs, I see a lot of "Slow Kaha DB access" logs. Some examples:

1. Slow KahaDB access: cleanup took 5138
2. Slow KahaDB access: Journal append took: 1635 ms, Index update took 2330 ms

This is a big issue in our system because as soon as AMQ stops responding quick enough, we lock up threads. It seems like becasue IO/access is taking so long, activeMQ stops responding to our ESBs. Since we continue to try to queue messages on ActiveMQ (intended functionality), we open more and more connections, using more and more threads till the threads are maxed out.

After a few minutes threads our released and activeMQ becomes responsive again, but by then its too late for our system because the ESBs spin out of control due to backed up traffic and activeMQ freezing up.

Anyone face the same problem? Any info anyone can provide on how to trouble shoot this is appreciated.

Thanks

Answer 1

Unless you absolutely require persistence, you can try sending non-persistent messages and change the broker to be non-persistent. See the ActiveMQ documentation on the precise steps to do that. In that case, the broker will not write any messages out to disk, but keep them only in memory. If the broker crashes however, that means that those messages are lost.

Otherwise, I'd recommend that you either address the IO performance directly by replacing the hardware, or by switching to a clustered ActiveMQ deployment to spread the load more evenly.