Set IAM role to run AWS CLI Python subprocess in AWS Lambda

Question

I created a Lambda layer that contains the AWS CLI because the aws sync command is not available in boto3 and the copy function was too slow to complete the s3 to s3 transfer prior to hitting the maximum Lambda time out.

I can run AWS CLI commands (ls, sync, etc) as the execution role for the Lambda. I need to run aws sync from Lambda to sync two s3 buckets as a different IAM user that has access to the source and destination buckets.

How do I configure the CLI to run as a particular user?

The commands below configure the user successfully locally but when run using the subprocess function in Lambda, the output of get-caller-identity is still the ARN of the Lambda role.

/opt/aws configure set aws_access_key_id <id>
/opt/aws configure set aws_secret_access_key <key>
/opt/aws sts get-caller-identity

In order to run the configure set id and key commands, I had to set an environmental variable to bypass the read only Lambda file system to write the credentials to the /temp directory.

Environmental Variable:

key: HOME
value: /tmp

Note: There is a populated credentials file written to: /temp/.aws/credentials

Lambda:

import logging
import subprocess

logger = logging.getLogger()
logger.setLevel(logging.INFO)

def run_command(command):
    command_list = command.split(' ')
    try:
        logger.info("Running shell command: \"{}\"".format(command))
        result = subprocess.run(command_list, stdout=subprocess.PIPE);
        logger.info("Command output:\n---\n{}\n---".format(result.stdout.decode('UTF-8')))
    except Exception as e:
        logger.error("Exception: {}".format(e))
        return False

    return True

def lambda_handler(event, context):
    run_command('/opt/aws configure set aws_access_key_id <id>')
    run_command('/opt/aws configure set aws_secret_access_key <key>')
    # not set to new IAM user
    run_command('/opt/aws sts get-caller-identity')

Thanks for reading!

Answer 1

Pass the IAM access credentials as environment variables to the subprocess, instead of trying to run aws configure :

result = subprocess.run(command_list, stdout=subprocess.PIPE, env={
  'AWS_ACCESS_KEY_ID': <id>,
  'AWS_SECRET_ACCESS_KEY': <id>,
});

---

Explanation:

The Lambda runtime environment has the environment variables set for the role it is using, documented here , and those environment variables are taking precedence over anything you are trying to do via the aws configure command. So the easiest way to fix that is to just override the environment variables that are passed to the subprocess via the env argument.

I don't think your calls to aws configure are working anyway, since all that does is write to a ~/.aws/credentials file, and that file isn't writeable inside the AWS Lambda environment, only /tmp is writeable.