GCP - how to add alert on number of messages sent to a pubsub dead letter queue?

Question

I have my application which processes messages from a pubsub topic and if it fails the message is send to a separate dlq topic. I want to be able to set an alarm in monitoring that when during a day there were 30k messages sent to the dlq it notifies me and I can check why my service is not wokring.

I tried to set up some polices in gcp but I don't know and couldn't find anywhere in the docs how to setup a metric of daily processed messages on a topic. Can anyone help me?

Answer 1

You can create a new alert policy like this

PubSub subscription/unacked messages.

You can add a filter on your subscription name if you have several subscriptions in your project.

Add the notification channel that you want, an email in my case. After few minutes, you can see the first alert

And the email

EDIT

For the acked messages, you can do this

I never tried an aggregation over 1 day, but it should be OK.

Answer 2

Please check the following GCP community tutorials which outline how to create an alert-based event archiver with Stackdriver and Cloud Pub/Sub

https://cloud.google.com/community/tutorials/cloud-pubsub-drainer