Starting EC2 instance with boto3 in a Chalice app is not working

Question

I'm having an issue using boto3 to start EC2 instances from a Lambda deployed by Chalice .

The relevant code is this:

resource = boto3.resource('ec2')
instance = resource.Instance(params['instance_id'])
if params['action'] == 'run':
    try:
        response = instance.start()
    except BaseException as be:
        logging.exception("Error: Failed to start instance" + str(be) )
        raise ChaliceViewError("Internal error at server side")
else:
    try:
        response = instance.stop(Force=True)
    except BaseException as be:
        logging.exception("Error: Failed to stop instance" + str(be) )
        raise ChaliceViewError("Internal error at server side")

The request appears to succeed. For instance, in 2 cases where the "start()" method was called the boto3 response was this: {"Status":{"StartingInstances":[{"CurrentState":{"Code":0,"Name":"pending"},"InstanceId":"i-0129bb4079559e5bc","PreviousState":{"Code":80,"Name":"stopped"}}],"ResponseMetadata":{"RequestId":"d88a9fbc-f2f2-4c51-9629-30a63c7e753b","HTTPStatusCode":200,"HTTPHeaders":{"x-amzn-requestid":"d88a9fbc-f2f2-4c51-9629-30a63c7e753b","content-type":"text/xml;charset=UTF-8","content-length":"579","date":"Wed, 23 Sep 2020 16:59:40 GMT","server":"AmazonEC2"},"RetryAttempts":0}}}

The other response is this:

{"Status":{"StartingInstances":[{"CurrentState":{"Code":0,"Name":"pending"},"InstanceId":"i-0129bb4079559e5bc","PreviousState":{"Code":80,"Name":"stopped"}}],"ResponseMetadata":{"RequestId":"2bde553a-87f1-4fe0-a13a-8b4db4c0dbbc","HTTPStatusCode":200,"HTTPHeaders":{"x-amzn-requestid":"2bde553a-87f1-4fe0-a13a-8b4db4c0dbbc","content-type":"text/xml;charset=UTF-8","content-length":"579","date":"Wed, 23 Sep 2020 17:07:58 GMT","server":"AmazonEC2"},"RetryAttempts":0}}}

However in both cases the instance did not start, the instance state in the AWS Console stayed at "stopped".

When I tried the same code snippet in a python console, it worked, and the instance started successfully:

>>> import boto3
>>> resource = boto3.resource('ec2')
>>> instance = resource.Instance('i-0129bb4079559e5bc')
>>> response = instance.start()
>>> response
{'StartingInstances': [{'CurrentState': {'Code': 0, 'Name': 'pending'}, 'InstanceId': 'i-0129bb4079559e5bc', 'PreviousState': {'Code': 80, 'Name': 'stopped'}}], 'ResponseMetadata': {'RequestId': '535224cc-21d8-45fa-a4a2-0ac984cdfe9a', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '535224cc-21d8-45fa-a4a2-0ac984cdfe9a', 'content-type': 'text/xml;charset=UTF-8', 'content-length': '579', 'date': 'Wed, 23 Sep 2020 17:05:10 GMT', 'server': 'AmazonEC2'}, 'RetryAttempts': 0}}

Has anyone seen this behavior before? Is there something obvious I'm missing?

Answer 1

I ended up going to AWS support with this question.

The machines that I was trying to start had been migrated from another AWS account and had their backing EBS volumes encrypted using a KMS key. The Lambda execution role needs access to use the KMS key in order to start the EC2 instance.

At the suggestion of the AWS tech, I added this statement to the KMS Key policy:

{
   "Sid": "Allow Lambda role use of the CMK",
   "Effect": "Allow",
   "Principal": {
       "AWS": [
           "<REPLACE WITH LAMBDA-EXECUTION-ROLE-ARN>"
       ]
   },
   "Action": [
       "kms:Encrypt",
       "kms:Decrypt",
       "kms:ReEncrypt*",
       "kms:GenerateDataKey*",
       "kms:DescribeKey",
       "kms:CreateGrant"
   ],
   "Resource": "*"
}

Once this was in place, the instance started successfully.

The one outstanding question I have (and I will update this answer if I receive it) is why the boto3 start operation returned a success if the Lambda didn't have permissions.