Metric for Number of unacknowledged messages older than 20 minutes

Question

I am trying to set up alerts on pubsub in gcp that monitor the number of old messages in a queue. Specifically the number of unacknowledged messages older than 20 minutes.

I want an alert that because number of unacknowledged messages cloud shoot high on sudden push of hugh number of messages. And using only the oldest unacknowledged message will run the alert for outlier messages that might stuck in the queue (ex bad formatted messages etc..)

I've tried to combine both metrics but could not know how to filter on one of them.

fetch pubsub_subscription |
{
    t_0: metric 'pubsub.googleapis.com/subscription/num_undelivered_messages';
    t_1: metric 'pubsub.googleapis.com/subscription/oldest_unacked_message_age'
}
| outer_join 0 # how to filter now on oldest_unacked_message_age > 20 minutes and select num_undelivered_messages

Also I think this won't work as my understanding of cloud pubsub metrics because each metric is a single time series number. It does not have information about individual messages (correct me if I am wrong).

Also I've tried to look for a metic that have them both but can't find one as well.

Answer 1

You can deploy an alert of undelivered messages in Google Cloud Monitoring . You will find the Pub/Sub subscription resource type, and then you can set a filter based on the response_code . Also, you can create a new chart based on your needs.