GCP PubSub difference between undelivered messages and dead letter messages

Question

I am creating alert systems to check dead message queues and in this GCP Document I see there are two metrics subscription/num_undelivered_messages and subscription/dead_letter_message_count . I find it hard to understand the difference since I assume undelivered messages are being published to the dead message queue. If that's the case, aren't these metrics reporting the same number?

Answer 1

You have to understand the flow to understand the metrics.

When a message arrive in the Subscription, it is undelivered. It stays in that state up to receiving a ACK message.

After several unsuccessful (nack, error, timeout,...) deliveries (at least 5, you can set lower than 5), your message can be delivered to a Dead Letter topic (if configured).

The undelivered messages can stay minutes, hours or days in the subscription, before being delivered to the Dead letter topic.

---

That being said, now what do you want to monitor?

• The number of messages that are stacked in your subscription to understand the backlogs that you have to process?
• The number of messages pushed to Dead Letter topic because there were a definitive error, and all the retries were unsuccessful?

Depends on your use case, but the ratio, the speed, the frequency of those metrics can indicate the health of your app.

Golden signals:

• error rate: Number of message in the Dead Letter topic
• Traffic: Number of message undelivered