AWS Resource Usage Data - CPU, Memory and Disk

Question

I am trying to build an analytics Dashboard using the below Metrics/KPIs for all the EC2 Instance.

1. Total CPU vs CPUUtilized
2. Total RAM vs RAMUtilized
3. Total EBS Volume vs EBSUtilized.

For example, I have lunch an EC2 instance with 4 CPU, 16GiB RAM and 50GB SSD, I would like to know the above KPIs in a time series trend. I am not getting any clue on where to get the data from EC2. Tried the EC2 instance metrics through CloudWatch using boto3 client, however did not get the above Metrics. I would like to know:

1. Where to find the data with above Metrics?
2. Need the above metrics data in s3 on an daily basis.

Similarly is there a way to get similar metrics for AWS RDS and AWS EKS Cluster?

Thanks!

Answer 1

The Amazon EC2 service collects information about the virtual machine (instance) and sends it to Amazon CloudWatch Logs.

See: List the available CloudWatch metrics for your instances - Amazon Elastic Compute Cloud

Note that it only collects metrics that can be observed from the virtual machine itself -- CPU Utilization,.network traffic and Amazon EBS traffic. The EC2 service cannot see what is happening 'inside' the instance, since it is the Operating System that controls memory and manages the contents of the disks .

If you wish to collect metrics from the Operating System, then you would need to Collect metrics and logs from Amazon EC2 instances and on-premises servers with the CloudWatch agent - Amazon CloudWatch . This agent runs in the instance and sends metrics out to CloudWatch.

You can write code that calls the CloudWatch Metrics APIs to retrieve metrics. Note that the metrics returned are calculated over a time period (eg average CPU Utilization over a 5-minute period). It is not possible to retrieve the actual raw datapoints.

See also:

• Monitoring Amazon RDS metrics with Amazon CloudWatch - Amazon Relational Database Service
• Amazon EKS and Kube.netes Container Insights metrics - Amazon CloudWatch