Cannot import sqlalchemy within AWS Lambda function

Question

I know the concept of using a deployment package is relatively straightforward, but I've been banging my head on this issue for the last few hours. I am following the documentation from AWS on packaging up Lambda dependencies . I want to write a simple Lambda function to update an entry in a PostgreSQL table upon some event.

I first make a new directory to work in:

mkdir lambdas-deployment && cd lambdas-deployment

Then I make a new virtual environment and install my packages:

virtualenv v-env
source v-env/bin/activate
pip3 install sqlalchemy boto3 psycopg2

My trigger-yaml-parse.py function (it doesn't actually use the sqlalchemy library yet, but I'm just trying to import it successfully):

import logging
import json
import boto3
import sqlalchemy

def lambda_handler(event, context):
    records = event['Records']
    s3_records = filter(lambda record: record['eventSource'] == 'aws:s3', records)
    object_created_records = filter(lambda record: record['eventName'].startswith('ObjectCreated'), s3_records)

    for record in object_created_records:
        key = record['s3']['object']['key']
        print(key)

I've been following the instructions in the AWS documentation.

zip -r trigger-yaml-parse.zip $VIRTUAL_ENV/lib/python3.6/site-packages/

I then add in my function code:

zip -g trigger-yaml-parse.zip trigger-yaml-parse.py

I get an output of updating: trigger-yaml-parse.py (deflated 48%) .

Then I upload my new zipped deployment to my S3 build bucket:

aws s3 cp trigger-yaml-parse.zip s3://lambda-build-bucket

I choose upload from S3 in the AWS Lambda console:

However, my Lambda function fails upon execution with the error:

START RequestId: 396c6c3c-3f5b-4df9-b7f1-057842a87eb3 Version: $LATEST
Unable to import module 'trigger-yaml-parse': No module named 'sqlalchemy'

What am I doing wrong? I've followed the documentation from AWS literally step for step.

Answer 1

from AWS documentation :

"Zip packages uploaded with incorrect permissions may cause execution failure. AWS Lambda requires global read permissions on code files and any dependent libraries that comprise your deployment package"

So you can use zip info to check permissions:

zipinfo trigger-yaml-parse.zip

-r-------- means only the file owner has permissions.

Answer 2

I think your problem might be in this line:

zip -r trigger-yaml-parse.zip $VIRTUAL_ENV/lib/python3.6/site-packages/

When you create the zip file the compressed files will have the complete path you had in your disk. The python runtime in lambda will not be able to find the libraries.

Instead you should do something like this

cd $VIRTUAL_ENV/lib/python3.6/site-packages/
zip -r /full/path/to/trigger-yaml-parse.zip .

Run unzip -t against both files and you will see the difference.