Utilizing custom metrics and rollback criteria for an AWS Elasticsearch cluster deployment

Question

Some configuration changes utilize a Blue/Green deployment for AWS Elasticsearch. These deployments can take a long time the Blue/Green deployment is meant to protect you from bad deployments and give quick rollback. Recently my team had a deployment that lead to issues and the recovery took a long time because the standard blue/green deployment by AWS didn't detect our issue (which is understandable because these metrics weren't related to the cluster itself)

Broadly my questions boil down to

What metrics is AWS looking at to determine if a deployment is bad and should be rolled back?

and

Can you instrument custrom metrics and rollback criteria to indicate to AWS that a deployment is bad outside of the standard cluster health metrics?

Answer 1

What metrics is AWS looking at to determine if a deployment is bad and should be rolled back?

These are potentially infrastructure level checks. Suppose you scaled up your cluster in a region but there are no free nodes available in that particular region for the instance type you selected. In this case you will continue to use the old (blue) nodes. There would be many other infrastructure level checks internal to Amazon but I guess you can understand that is at infrastructure level.

Can you instrument custom metrics and rollback criteria to indicate to AWS that a deployment is bad outside of the standard cluster health metrics?

No. Since AWS Elasticsearch is full managed service, the blue-green deployment is completely internal and users can only monitor it but cannot define and custom metrics or cases for rollback.