How to set a GCP Cloud Monitoring (Stackdriver) alert policy period greater than 24 hours?

Question

Currently 24 hours is the limit of time a Cloud Monitoring (erstwhile Stackdriver) alert policy can be set.

However, if you have a daily activity, like a database backup, it might take a little more or less time each day (eg run in 1 hour 10min one day, 1 hour 12min the next day). In this case, you might not see your completion indicator until 24 hours and 2 minutes since the prior indicator. This will cause Cloud Monitoring to issue an alert (because you are +2min over the alerting window limit).

Is there a way to better handle the variance in these alerts, like a 25 hour look back period?

Answer 1

Currently, there is no way to increase the period time over 24 hours.

However, there is a Feature Request already opened for that.

You can follow it in this public link [1].

Cheers,

---

[1] https://issuetracker.google.com/175703606

Answer 2

I found a work around to this problem.

1. Create a metric for when your job starts (eg started_metric )
2. Create a metric for when your job finishes (eg completed_metric )

Now create a two part Alert Policy

1. Require that started_metric occurs once per 24 hours
2. Require that completed_metric occurs once per 24 hours
3. Trigger if (1) and (2) above are met (eg both > 24 hours)

This works around the 24 hour job jitter issue, as the job might take > 24 hours to complete, but it should always start (eg cron job) within 24 hours.