Cloudwatch: Monitoring SNS based on MessageAttribute?

Question

. 。 Answers to this question are eligible for a +50 reputation bounty. Yaakov Bressler wants to to this question: ：

A suitable answer addresses capabilities of Cloudwatch and provides a solution. ("It is not possible" is not a solution here.) I imagine a workaround is required – if this is so, please provide a workaround which abides by cloud best practices + is cost optimized.

(Also suitable answers are "here's how I solved it, here are the tradeoffs to that solution.")

I have several backend services which publish to a single SNS topic. The specific publishing service is designated as a field in MessageAttributes . Conversely, I have a single consumer subscribed to all messages from this topic.

I'd like to monitor this SNS topic in Cloudwatch to determine if the number of messages being published per backend service is as expected. Is there a way for me to partition this information in cloudwatch based on a value in message attributes?

Note: I am aware that Cloudwatch can monitor number of messages based on SNS Filter Policies . My concern is that I have a single consumer – would creating additional filter policies be expensive/wasteful?

Answer 1

The filter policy metrics sadly do not support the granularity you are looking for at a policy level. In order to keep the metrics within Cloudwatch you could use a Lambda that processes the messages on the topic and generates custom metrics into Cloudwatch based on the Attribute you are interested in making it possible to use in dashboards and alarms.

At a high-level the flow would be SNS -> Lambda -> Cloudwatch. Rather than using the CloudWatch API to push the metrics you could make use of the Cloudwatch embedded metric format.

Depending on your setup you could instead add metrics into the backend services that are making the PublishMessage calls. At least this way no new components would be needed to capture the metrics.