Apache Camel immediate read from SQS

Question

I have very huge camel pipeline which starts from consuming message from SQS. The time requires for whole process is vary. From 5 sec to 30 min, it is hard to guess here.

What I want to achieve:

1. Do not guess visibility timeout size, and just delete message from SQS as soon as message is consumed.

What I already tried:

1. Tried Camel option deleteAfterRead=true -> doesn't help, because as stated in the doc: Delete message from SQS after it has been read (and processed by the route). And I have huge pipeline. So processed requirement fails here.
2. Tried to increase visibility timeout, but as I stated, it is just a guessing game, and I want to develop more reliable solution.

Thank you for help!

Answer 1

You should certainly NOT delete the message as soon as you consume it if you are not finished processing it because you will lose the message if your application crashes.

What you should do instead is extend the visibility timeout manually if you realize that your processing time gets close to the original visibility timeout. Not sure how you would implement it in Camel but we previously integrated into the default springframework.cloud.aws.messaging : https://github.com/Mercateo/sqs-utils

Answer 2

One option is to add an exception handler and mark the exception as handled calling handled(true) on the OnExceptionDefinition .

Another workaround is to send the message to a wireTap and do your processing there. This ends the route and causes a deleteMessage to be sent to the queue.

I myself am still looking for a other options tho.