How should I pass my s3 credentials to Python lambda function on AWS?

Question

I'd like to write a file to S3 from my lambda function written in Python. But I'm struggling to pass my S3 ID and Key.

The following works on my local machine after I set my local Python environment variables AWS_SHARED_CREDENTIALS_FILE and AWS_CONFIG_FILE to point to the local files I created with the AWS CLI.

session = boto3.session.Session(region_name='us-east-2') 
s3 = session.client('s3', 
     config=boto3.session.Config(signature_version='s3v4'))

And the following works on Lambda where I hand code my ID and Key (using *** here):

AWS_ACCESS_KEY_ID = '***'
AWS_SECRET_ACCESS_KEY = '***'
session = boto3.session.Session(region_name='us-east-2') 
s3 = session.client('s3', 
     config=boto3.session.Config(signature_version='s3v4'),
     aws_access_key_id=AWS_ACCESS_KEY_ID,
     aws_secret_access_key=AWS_SECRET_ACCESS_KEY)

But I understand this is insecure after reading best practices from Amazon. So I try:

AWS_ACCESS_KEY_ID = os.environ['AWS_ACCESS_KEY_ID']
AWS_SECRET_ACCESS_KEY = os.environ['AWS_SECRET_ACCESS_KEY']
session = boto3.session.Session(region_name='us-east-2') 
s3 = session.client('s3', 
     config=boto3.session.Config(signature_version='s3v4'),
     aws_access_key_id=AWS_ACCESS_KEY_ID,
     aws_secret_access_key=AWS_SECRET_ACCESS_KEY)

But I get an error: “The AWS Access Key Id you provided does not exist in our records.” I also tried to define these variables in the Lambda console, but I then I get: "Lambda was unable to configure your environment variables because the environment variables you have provided contains reserved keys."

I am a little surprised I need to pass an ID or Key at all since I believe my account for authoring the Lambda function also has permission to write to the S3 account (the key and secret I hand code are from IAM for this same account). I got the same sense from reading the following post: AWS Lambda function write to S3

Answer 1

The fact that your account has permissions to "write" to the S3 account does not mean that you can do it. AWS has a service called IAM which will handle the permissions that your lambda (among many other services) have to perform actions against other AWS resources.

Most probably you are lacking the relevant IAM role/policy associated to your lambda to write to the S3 bucket.

As pointed in AWS Lambda Permissions Model you need to create and associate an IAM role when creating the lambda. You can do this from the console or using CloudFormation .

Once you have the relevant permissions in place for your lambda you do not need to deal with keys or authentication.

Answer 2

You never need to use AWS access keys when you are using one AWS resource within another. Just allow the Lambda function to access the S3 bucket and any actions that you want to take (eg PutObject). If you make sure the Lambda function receives the role with the policy to allow for that kind of access, the SDK takes all authentication out of your hands.

If you do need to use any secrets keys in Lambda, eg of 3rd party systems or an AWS RDS database (non-Aurora), you may want to have a look at AWS KMS. This works nicely together with Lambda. But again: using S3 in Lambda should be handled with the right role/policy in IAM.

Answer 3

Incidentally, I was curious to see if it took less time for AWS to authenticate me if I explicitly passed ACCESS_KEY and SECRET_ACCESS_KEY when accessing AWS DynamoDB from AWS Lambda instead of letting Lambda find my credentials automatically via environment variables. Here's the log entry from AWS Cloudwatch for example:

[INFO] 2018-12-23T15:34:56.174Z 4c399add-06c8-11e9-8970-3b3cbb83cd9c Found credentials in environment variables.

I did this because when I switched from using AWS RDS with MySQL to DynamoDB it took nearly 1000ms longer to complete some simple table reads and updates. For reference, my calls to AWS RDS MySQL passed credentials explicitly eg:

conn = pymysql.connect(host=db_host, port=db_port, user=db_user, \
                            passwd=db_pass, db=db_name, connect_timeout=5)

so, I thought this might be the problem because when connecting to DynamoDB I was using:

db = boto3.resource('dynamodb')
table = db.Table(dynamo_db_table)

I decided to try the following to see if my authentication time decreased:

session = boto3.Session(
            aws_access_key_id=AWS_ACCESS_KEY_ID,
            aws_secret_access_key=AWS_SECRET_ACCESS_KEY
            )

db = session.resource('dynamodb')
table = db.Table(dynamo_db_table)

End result was explicitly providing my access keys ended up saving less than 100ms on average so I went back to letting the Lambda execution environment dynamically determine my access credential. Still working to understand why MySQL is so much faster for my simple table {key:value} query and update use case.