aggregating codahale metrics counts across ECS task instances in CloudWatch

Question

I've got an ECS service reporting metrics to CloudWatch collected with Codahale Metrics. Some of the metrics are counts, eg count of requests made to an external service. Each service instance maintains and reports to CloudWatch its own count. To my understanding it means the values of the count in CloudWatch are the individuals counts per service without a possibility to see eg the total. If each instance was making 300 requests than the value reported would be 300, with not way to sum it up to 900.

What is the best way to fix it? Is adding an additional dimension with eg ecs task id to the reported CloudWatch metric the way?

I'm graphing the results in Grafana, but likely it's not the important part.

Answer 1

Metrics are already aggregated in Cloudwatch assuming they have the same namespace and name. If these service request metrics are the same, they should be the same metric, then you can add Dimensions to them, such as TaskId , RequestedService or whatever you wanted to aggregate by.

Typically you have the opposite challenge in Cloudwatch Metrics to what you are describing. Metrics are already aggregated together and then you want to drill down to a particular values to debug some issue, such as if you had a problem with a particular container task you would set the dimension TaskId=todo1 , or if you suspected a service is down you'd set RequestedService=todo2 .

I suspect you are creating a metric for each service you make requests to, instead you only want one metric, and add dimensions to it as described earlier.

Also for this particular use-case you might want to consider open-telemetry/X-Ray which will create for you a service graph and handles the specific case of tracing requests through different services. That does take a bit of effort to setup though.