One or more points were written more frequently than the maximum sampling period configured for the metric

Question

Background

I have a website deployed in multiple machines. I want to create a Google Custom Metric that specifies the throughput of it - how many calls were served.

The idea was to create a custom metric that collects information about served requests and 1 time per minute to update the information into a custom metric. So, for each machine, this code can happen a maximum of 1-time per minute. But this process is happening on each machine on my cluster.

Running the code locally is working perfectly.

The problem

I'm getting this error: Grpc.Core.RpcException: Status(StatusCode=InvalidArgument, Detail="One or more TimeSeries could not be written: One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/web/2xx, Timestamps: {Youngest Existing: '2019/09/28-23:58:59.000', New: '2019/09/28-23:59:02.000'}}: timeSeries[0]; One or more points were written more frequently than the maximum sampling period configured for the metric. {Metric: custom.googleapis.com/web/4xx, Timestamps: {Youngest Existing: '2019/09/28-23:58:59.000', New: '2019/09/28-23:59:02.000'}}: timeSeries 1 ")

Then, I was reading in the custom metric limitations that:

Rate at which data can be written to a single time series = one point per minute

I was thinking that Google Cloud Custom Metric will handle the concurrencies issues for me.

According to their limitations, the only option for me to implement realtime monitoring is to put another application that will collect information from all machines and will update it into a custom metric. It sounds to me like too much work for a real use case.

What I'm missing?

Answer 1

Now that you add the machine name on the metric and you get the machines metrics.

To SUM these metrics go to Stackdriver > Metric Explorer, and group your metrics by project-id or label for example, and then SUM the metrics.

https://cloud.google.com/monitoring/charts/metrics-selector#alignment

You can save the chart in a custom dashboard.