Which of these MQ Consumer Designs is better?

Question

Situation

In my case, there is one MQ server for queue up jobs. After the job completion, the job result is required to write to database. I will need to write one or more programs to dequeue the jobs, process the job and write the job results to database. I have a few design in my mind but I am not sure which one I should pick nor the pros and cons of the designs.

There are some requirements though, the program should try to ensure:

• No duplicated processing of the job when the server (assume the job is sending email, you don't want the customer to receive 2 emails with the same content)
• No lost of job result (so that you know the job result from querying the database).

Performance is not much concern in my case, because processing one job will probably takes at least 10 seconds. However I still want a good design so that I can reuse it later when performance is a concern.

Design 1

A number of threads will be created and working individually. Each thread will dequeue a job from the MQ server, process the job and write the job result to database. After that, the thread loop again to dequeue job.

The number of session to MQ server and number of connection to database server will be the same as number of threads. This should be an issue if performance is a concern.

Design 2

One dispatcher thread is created to dequeue jobs from MQ server and dispatch to a worker thread pool. Each worker thread work individually to process the job and write the job result to database.

Design 3

One dispatcher thread is created to dequeue jobs from MQ server and dispatch to a worker thread pool. Each worker thread work individually to process the job. After the job completed the worker thread will be passed the job result to another queue in the MQ server. Another database thread will dequeue the job result from MQ server and write the result to database.

This design can help if the database server is down, so that the program is still able to process the job without losing job result.

Design 4

A number of threads will be created and working individually. Each thread will dequeue a job from the MQ server and process the job. After the job completed the worker thread will be passed the job result to another queue in the MQ server. A number of database threads will dequeue the job result from MQ server and write the result to database.

To me, this looks like the best design which avoid the complex design of dispatcher and able to take the advantage to queue up the job result when database server is down.

Other Designs

There are some more designs can think of. They are similar to above 3 designs but only apply to writing the job result to database part. Therefore I am not going to list again.

Remark

This is my first time to work on MQ server project.

Answer 1

I think you need to read up on IBM MQ (aka WebSphere MQ). MQ does not handle 'jobs' - it handles messages. MQ is 'once and only once' delivery of messages and it has assured delivery (for persistent messages).

Designs #2 & 3 are a very bad idea. You are decoupling the process and if either the dispatcher or the worker thread crashes (WHICH HAPPENS) then the message will be lost. And of course the application support team will blame MQ because they cannot blame themselves for creating a horrible system.

Design # 1 is good so long as you use two-phase commit (read about MQ ETC).

If your database has outages then use design #4. Use single-phase commit for the MQ thread and two-phase commit for the database thread.