CloudWatch Monitoring and Notifications

Question

I am using various AWS services (Lambda, Glue, S3, Redshift, EC2) for ETL processing. I am trying to create a 'log ETL flow' to have monitoring and notifications sent out (email or otherwise) when a step fails in the process.

I have checked that each service I am using has metrics being logged in CloudWatch. I am now trying to figure out a smart way of processing this data in order to send out notifications when a step fails and/or have a central monitoring of the entire flow.

Are there any best practices or examples of this setup?

Answer 1

https://docs.amazonaws.cn/en_us/AmazonCloudWatch/latest/monitoring/CloudWatch-Usage-Metrics.html

You can find the best practices listed on the page.

Answer 2

It seems to be the perfect case for the CloudWatch Alarms .

You can create a CloudWatch alarm that watches a single CloudWatch metric or the result of a math expression based on CloudWatch metrics. The alarm performs one or more actions based on the value of the metric or expression relative to a threshold over a number of time periods. The action can be an Amazon EC2 action, an Amazon EC2 Auto Scaling action, or a notification sent to an Amazon SNS topic.

You can create a chain CloudWatch Alarm -> SNS

You can either use SNS to notify users via SMS or Push Notifications .
Or you can do one step more SNS -> SES to deliver emails.