在PUB / SUB模型中使用RabbitMQ的消息

Question

In my project i have a requirement like different methods push sql queries to Rabbit queue (basically an exchange which broadcast messages to all queues). These SQL queries pushed to queue are not very important at this point of time (basically for reporting purpose) when compare to production, so i will be executing these queries in different environment (some other standalone project) when i dont have much load on server.

Now lets get into part two of the story, In standalone application at other side have 2 consumers which consume messages (SQL queries) from RabbitMQ (exchange). One consumer always logs (like backup of queries) and other performs batch update/insert on DB.

My issue is what is the good way to achieve this use case when few queries may fail because of several reasons and as it is batch operation , that entire batch is going to rollback and i cant get those from queue again as those were already consumed. I know that i have to get it from backup created by my 2nd consumer, but i have to stop 1st consumer after getting any exception and what is the good way of fetching that particular failed query from backup log (consider lacs of queries, am thinking of using rolling appender log for multi log files). Thanks for patiently understanding my problem and please suggest a good way of achieving this issue.

Answer 1

try to categorize the failures that might occur on the consumer side. There might be

• permanent errors (such as invalid SQL statements) and
• temporary errors (network glitches, database server gone/restart,...) etc.

for temporary errors just retry the processing of the message (see spring's retry library). messages that result into permanent errors might go into an invalid letter queue for further analysis/reporting (add the exception as header to the message before rerouting)