Java/JMS - Processing number of messages with handling failure scenarios

Question

I have a requirement where I have to process number of messages. These messages will be inserted into a DB table by another process. All I have to do is to check for new messages in DB and send it to corresponding customer either email or http, depending on their configuration. The number of messages could be several thousands at any given time and the number of customers are around 1000.

I am planning to design this the way producer consumer works. Like producer thread polls DB for new messages and puts them into Queue and worker threads read these messages and processes.

At first it looked like JMS is the right solution for this requirement. But am seeking if any better alternatively like ExecutorService, using Threadpool suitable for this requirement given the following scenarios.

1. if one message delivery fails, I will have to retry it several times at least for 24 hours.
2. if one message delivery fails for a customer, all chances that other message delivery will also fail. So, I should try to send the first message before I process next message for that customer.

That means if I have one queue for all messages for all customers, then if one message fails, others will not be processed.

Can someone give me suggestion on how best I can handle this?

Thanks in advance.

Answer 1

You should take a serious look at Apache Camel . It handles all of the threading, producing & consuming for you with a large number of built in components (SQL, JMS, SMTP, HTTP, File, etc) and your terminology around Consumers/Producers matches the Camel view of integration.

Implementing something like you describe is as easy as:

from("sql:select * from table")
  .to("jms:myQueue");

Have a look at the list of endpoints

Answer 2

Retry logic is relatively easy to implement with generic JMS: no transactions, assign message a retry attribute, increment it with each retry, pause in listener when retrying, quit after limit is exhausted.

Queue per customer is the simplest configuration. Dynamic queues may help you with customers: create a queue and a listener for each customer dynamically.

Actually polling database reliably may present more challenge for you.