AWS beanstalk monitoring metrics disappear during deploy

Question

We have setup an immutable configuration for AWS beanstalk using both web and worker classes (multiple clusters). When we deploy a new application, it creates a temporary autoscaling group and then deploys to that and in end switches back to the old autoscaling group. This process takes about 20-30 mins and works fine.

Although, every time we deploy the application the monitoring stats: CPU utilization, Memory utilization, disk space and so on, all disappear for 5-6 hours before returning back. Seems like an AWS issue but not sure if we are doing anything wrong. Has anyone else experience such behavior? Is there a workaround?

EDIT: Here's the screenshot for missing memory metric, as you can see there was no data between 5.30pm and 11.30pm.

Answer 1

I tried to replicate the issue using load-balanced EB environment with one instance by checking CPUUtilization .

I observed a small gap (10 minutes) after the immutable deployment. Which is far from 5-6 hours.

The observed delay was only in EB console . There was no delay in the corresponding CloudWatch (CW) metrics. Thus I could monitor the CPUUtilization in the CW while waiting for EB console to catch up.

For my tests I performed two immutable deployments . And in CW, the metrics align nicely (without any gaps) for the new instances created by the deployments:

Metrics for your instances should also be viable in CW. Thus you should be able to view them there, while EB consoles catches up.

To get the unified view of all the individual metrics, one can use metric math :

AVG(METRICS())

Answer 2

For anyone who finds this issue in future - I figured out the issue. The script I am using is provided by Cloudwatch that internally caches the auto-scaling group name for 6 hours. Typically that would not be an issue, since these values do not change regularly. But due to immutable deploy a temporary auto-scaling group is created during deploy that gets cached for 6 hours.

To resolve this, I replaced the following line in the CloudWatchClient code:

$meta_data_short_ttl = 21600; # 6 hours

with:

meta_data_short_ttl = 600; # 10 mins

Can also be done via .ebextension change from existing code if you are using this script :

04-reduce-cache:
    command: sed -i 's/meta_data_short_ttl = 21600; # 6 hours/meta_data_short_ttl = 600; # 10 mins/g' /opt/cloudwatch/aws-scripts-mon/CloudWatchClient.pm