How to block AWS SQS FIFO while having a message in the deadletter queue?

Question

Imagine the following lifetime of an Order.

Order is Paid

Order is Approved

Order is Completed

We chose to use an SQS FIFO to ensure all these messages are processed in the order they are produced, to avoid for example changing the status of an order to Approved only after it was Paid and not after has been Completed.

But let's say that there is an error while trying to Approve an order, and after several attempts the message will be moved to the Deadletter queue.

The problem we noticed is the subsequent message, that is "Order is completed", it is processed, even though the previous message, "Approved", it is in the deadletter queue.

How we should handle this?

Should we check the contents of deadletter queue for having messages with the same MessageGroupID as the consuming one, assuming we could do this?

Is there a mechanism that we are missing?

Answer 1

You don't have to block the queue, but rather only add messages when they are good to be processed. If you have a flow in which certain steps (messages to be processed) depend on the success of previous steps, you should make these messages only be added after the previous step (message) succeeds.

Let's say you have an SQS queue with a handler to process the message, it could look something like below.

Since you're using the same FIFO queue for all steps, I'm using the STEP as the MessageGroupId to allow messages of different steps to be processed in parallel (as they could belong to different orders), but the steps of one particular order are always processed in sequence and require the previous one to succeed.

On a side note, you shouldn't need FIFO queues with the approach below, and you could have separate queues with separate handlers for each step.

const sqs = new AWS.SQS();

const STEPS = {
  ORDER_PAID: "ORDER_PAID",
  ORDER_APPROVED: "ORDER_APPROVED",
  ORDER_COMPLETED: "ORDER_COMPLETED",
};

async function sendMessage(step: string, orderId: string) {
  return sqs
    .sendMessage({
      QueueUrl: process.env.YOUR_QUEUE_URL || "",
      MessageGroupId: step,
      MessageBody: JSON.stringify({
        currentStep: step,
        orderId,
      }),
    })
    .promise();
}

exports.handler = async function (event: any, context: any) {
  for (const message of event.Records) {
    const { currentStep, orderId } = JSON.parse(message.body);
    if (currentStep === STEPS.ORDER_PAID) {
      // process order paid, then add next step back to queue
      await sendMessage(STEPS.ORDER_APPROVED, orderId);
    }
    if (currentStep === STEPS.ORDER_APPROVED) {
      // process order approved, then add next step back to queue
      await sendMessage(STEPS.ORDER_COMPLETED, orderId);
    }
    if (currentStep === STEPS.ORDER_COMPLETED) {
      // process order completed
    }
  }
  return context.logStreamName;
};