JMS Message Ordering and Transaction Rollback

Question

We're building a system that will be sending messages from one application to another via JMS (Using Websphere MQ if that matters). These messages are of the form "Create x" or "Delete x". (The end result of this is that a third-party system needs to be informed of the Create and Delete messages, so the Read end of the JMS queue is going to talk to the third-party system, whilst the Write end of the JMS queue is just broadcasting messages out to be handled)

The problem that we're worried about here is if one of the messages fails. The initial thought here was simply to roll the failures back onto the JMS queue and let the normal retry mechanism handle it. That works until you get a Delete followed by a Create for the same identifier, eg

• Delete 123 - Fails, gets rolled back on to the queue
• Create 123 - Succeeds
• Delete 123 - Retry from earlier failure

The end result of this is that the third party was told to Create 123 and then immediately to Delete 123, instead of the other way around.

Whilst not ideal, from what I've been reading up on it seems that Message Affinity would help here, so that we can guarantee that the messages are processed in the correct order. However, I'm not sure how message affinity will work when messages are processed and failed back onto the queue? (Message Affinity is generally considered a bad idea, but the load here isn't going to be great, and the risk of poison messages is very low. It's simply the risk that the third-party that we're interacting with has a brief outage that we're concerned with)

Failing that, are there any better thoughts on how to do this?

Edit - Further complications. The system we're building to integrate with the third-party is to replace a system they used from a different supplier until recently. As such, there's a bunch of data that is already in the third-party, but it turns out to be very difficult to actually get this out. (The third-party doesn't even send success/failure messages back, merely an acknowledgement of receipt!), so we don't actually know the initial state of the system.

Answer 1

The definitive way to address this is to include in the message a sequence such that the earlier message can't overwrite the later one.

Once upon a time, your transactions at the bank were processed in the order they arrived. However, it was this exact problem that caused that to change. People became aware that their account balance could be positively or negatively affected depending on the order in which the transactions were processed. When it was left to chance it was occasionally a problem for people but in general no malice was perceived.

Later, banks started to memo-post transactions during the day, then sort them into the order most favorable to the bank prior to processing them. For example, if the largest checks cleared first and the account ran out of money, several smaller checks might bounce, generating multiple bounce fees for the bank. Once this was discovered to have become widespread practice, it was changed to always apply the transactions in the order most favorable to the account holder. (At least here in the US.)

This is a common problem and it's been solved many times, the first being decades ago. There really is no excuse anymore for an Enterprise-grade application to both

1. Use asynchronous messaging in which delivery order by design cannot be guaranteed, and
2. Contain an inherent dependency on delivery order to assure the integrity of the transactions and data.

The mention of message affinities hints at the solution to this when approached as a transport problem. To guarantee message order delivery requires the following:

• One and only one sender of messages
• One and only one node from which messages are sent.
• One and only one path between sender and receiver of messages.
• One and only one queue at which messages are received.
• All messages processed under syncpoint.
• The ability to pend processing of any messages whilst an orphaned transaction exists (connection exception handling).
• No backout queue on the input queue.
• No DLQ if the messages traverse a channel.
• No use of Priority delivery on any queue along the path.
• One and only one receiver of messages.
• No ability to scale the application other than by adding CPU and memory to the node(s) hosting the QMgr(s).

Or the problem could be addressed in the application design using memo posting, compensating transactions, or any of the other techniques commonly used to eliminate sequence dependencies.

(Side note: If the application in question is an off-the-shelf vendor package this would seem to me to create a credibility issue. No application can claim to be robust if something so commonplace as a backed out message can mess with data integrity.)

Answer 2

The one way to avoid the scenario you described above is to have Different classification of message failures ; keeping in mind that your Message should be processed in order.(Message affinity)

Consider scenario :-

If application receives Delete X for which it haven't receive Create x before, then classify this Error scenario as " Business Error " since this Error occured because Producer of Message sent wrong message . Or we can say Producer of message sent message in wrong order.

Once you classify this error as " Business Error ", you should not call rollback; instead of that insert this message into Database with "Business Error" as identification.

So in this way, you committed this message from queue and reduced risk of rollback and further reduce risk of inconsistent behaviour of your application.

Now Consider Another Scenario :-

If your application itself has some problem,( like , database or web server goes down or any such technical error) then in that case use Rollback mechanism of JMS queue and treate this Error as " Technical Error ".

So if any " Technical Error " occured, JMS Queue will retry message untill your Application is able to accept and process those.

Once your application is up after this " Technical Error " and tried processing messages in sequential order,same rule applied here ie if "Business Error" occured then that message will no longer retried.

Note : The "Business Error" classification should be agreed by all parties, ie if you are marking any message as a " Business Error " it means this message is no longer useful and your Producer should sent a new "Delete x" for any Valid 'Create x".

Some of the "Business Error" you can take into accounts are--

1. Received "Delete x" before "Create x"

2. Received "Create x" after "Create x"

3. Received "Delete x" after valid "Delete x"