Alarm Action “Terminate EC2 Instance” failed

Question

when I start a certain type of instance the userdata-script creates a metric + alarm via Boto. The metric delivers its data to CloudWatch correctly. The alarm should terminate the instance as an action if some condition based on the metric matches. In CloudWatch the alarm seems to be created correctly and it switches the alarm-states as desired.

BUT: When it comes to execute the action it fails with the following "history" entry: Alarm updated from

• OK to ALARM. Reason: Threshold Crossed: 5 datapoints were greater than the threshold (200.0). The most recent datapoints: 999.0, 999.0.
• arn:aws:automate:eu-west-1:ec2:terminate is in progress.
• Terminate EC2 Instance 'i-xxx' action failed. AWS was not able to validate the provided access credentials.

I've already granted the policy "AdministratorAccess" to the "userdata"-Role which is attached to the instance.

Any hints?

Regards Tom

Answer 1

I believe this is the issue. From the developer guide :

If you are using an IAM role (eg, an Amazon EC2 instance profile), you cannot stop or terminate the instance using alarm actions. However, you can still see the alarm state and perform any other actions such as Amazon SNS notifications or Auto Scaling policies.

I recently posted in the AWS Forum about this issue myself: https://forums.aws.amazon.com/message.jspa?messageID=601951

Answer 2

I just occured this question and seems I have resolved it.
The CloudWatch Alarm's service-linked IAM role is AWSServiceRoleForCloudWatchEvents. I find its Trusted entities is events.amazonaws.com. And its policy document in the Trust relationships tag,

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

And my ec2 instance's role policy document in the Trust relationships tag

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

And then add the Trusted entities, events.amazonaws.com content to the ec2 instance's role policy document, as follow:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

It add an trusted entity of events.amazonaws.com to the role.
Then, the CloudWatch Alarm to stop instance function is OK!