When to shut down message processing in case of queue / database failure?

Question

This is more of a best practice question for the common case of an application receiving messages, persisting them to a database, and possibly sending messages as a result.

• Assume transactionality sorts out atomic commit; what is a good policy on when to shut the application down altogether?
• If the database fails, the application could get flooded by messages, which it will end up rejecting. Should it give up immediately?
• If the outbound messaging service fails, the database will be flooded with rollbacks. Again, is it best to give up immediately?

More brownie points for any hints on how to best force a spring app to shut down in this case, as the default listener contain will catch any runtime exception and keep running.

Answer 1

From what I understand you're looking for following:

1. Do not loose messages from inbound queue just because application cannot process them.
2. When to stop processing if errors occur during processing.

First of all it's important to analyze the infrastructure you're dealing with and the kind of situations you'll have to deal with. Typical down times and how often they occur in various tiers of the system. How reliable is the network, is you db a rac server etc.

1. JMS already provides for mechanisms of retry. If message processing fails, send it back to queue until retires expire. This makes sense only if coupled with a delay so that flooding doesn't occur. If a small delay will not affect the transaction, I would recommend using messages with a delay. depending on your JMS provider, this is supported custom to container. Using a dead letter or exception queue when message from inbound queue cannot be processed can help with loss of messages.

2. Again, you can be the best judge of situation. You can define a property as how many many consecutive sends to dead letter queue constitute a shut down condition. You can tweak it during your system test to avoid false positives.

Answer 2

As cracked_all also mentioned, it is not recommended to give up immediately.

I think the best way would be to have other databases ready to function as the primary fails. Upon receiving unsuccessful acknowledgment, you can route them to the secondary one. Therefore, you don't lose that much of data. For this case, you can use "Guaranteed Delivery" feature in JMS.

With Guaranteed Delivery, the messaging system uses a built-in data store to persist messages. Each computer the messaging system is installed on has its own data store so that the messages can be stored locally. When the sender sends a message, the send operation does not complete successfully until the message is safely stored in the sender's data store. Subsequently, the message is not deleted from one data store until it is successfully forwarded to and stored in the next data store. In this way, once the sender successfully sends the message, it is always stored on disk on at least one computer until is successfully delivered to and acknowledged by the receiver. 1