AWS Lambda with Python 3 and virtualenv failed to install dependencies

Question

I'm working on a project in Python3.6 and I use AWS Lambda to implement some functions in python. I have written a buildspec.yml file to "build" and deploy my function from a repository to lambda functions. Here is the code:

version: 0.2
phases:
 install:
   commands:
     - echo "install step"
     - apt-get update
     - apt-get install zip -y
     - apt-get install python3-pip -y
 pre_build:
   commands:
     - echo "pre_build step"
     - pip install --upgrade pip
     - pip install --upgrade awscli
     - pip install --upgrade virtualenv
     # Define directories
     - export HOME_DIR=`pwd`
     - export PREPROCESSING_DIR=$HOME_DIR/preprocessing
     - export COMPARE_DIR=$HOME_DIR/compareHilightGood
     - export LAUNCH_HILIGHT_DIR=$HOME_DIR/LaunchHiLight
     - export NLTK_DATA=$HOME_DIR/nltk_data
     - mkdir nltk_data
     # create virtual environements
     - cd $HOME_DIR
     - virtualenv venv_preprocessing
     - virtualenv venv_compare
     - export SITE_PACKAGE_PREPROCESSING=$HOME_DIR/venv_preprocessing/lib/python3.6/site-packages
     - export SITE_PACKAGE_COMPARE=$HOME_DIR/venv_compare/lib/python3.6/site-packages
 build:
   commands:
     - echo "build step"
     - cd $HOME_DIR
     # Configure preprocessing virtual environement
     - . venv_preprocessing/bin/activate
       pip install requests
       pip install nltk
       python -m nltk.downloader -d $NLTK_DATA wordnet stopwords punkt
       deactivate
     - mv $NLTK_DATA $SITE_PACKAGE_PREPROCESSING
     - mv $PREPROCESSING_DIR/* $SITE_PACKAGE_PREPROCESSING
     - cd $SITE_PACKAGE_PREPROCESSING
     - sudo zip -r9 -q $HOME_DIR/preprocessing.zip .
     # Configure compare virtual environement
     - cd $HOME_DIR
     - . venv_compare/bin/activate
       pip install gensim
       pip install pandas
       deactivate
     - mv $COMPARE_DIR/* $SITE_PACKAGE_COMPARE
     - cd $SITE_PACKAGE_COMPARE
     - sudo zip -r9 -q $HOME_DIR/compare.zip .
     # Launch hilight
     - cd $LAUNCH_HILIGHT_DIR
     - sudo zip -r9 -q $HOME_DIR/launchHilight.zip .
 post_build:
   commands:
     - echo "post_build step"
     - cd $HOME_DIR
     - ls
     # preprocessing
     - aws s3 rm s3://lambda-preprocessing --recursive
     - aws s3 cp --acl public-read preprocessing.zip s3://lambda-preprocessing/preprocessing.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --s3-bucket lambda-preprocessing --s3-key preprocessing.zip
     - aws lambda update-function-configuration --function-name arn:aws:lambda:eu-west-3:671560023774:function:preprocessing --environment 'Variables={NLTK_DATA=/var/task/nltk_data}'
     # compare hilight good
     - aws s3 rm s3://lambda-comparehilightgood --recursive
     - aws s3 cp --acl public-read compare.zip s3://lambda-comparehilightgood/compare.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:compareHilightGood --s3-bucket lambda-comparehilightgood --s3-key compare.zip
     # launchHilight
     - aws s3 rm s3://hilightalgo --recursive
     - aws s3 cp --quiet --acl public-read launchHilight.zip s3://hilightalgo/launchHilight.zip
     - aws lambda update-function-code --function-name arn:aws:lambda:eu-west-3:671560023774:function:LaunchHilight --s3-bucket hilightalgo --s3-key launchHilight.zip
artifacts:
  files:
    - '**/*'

In this build process, I create two virtualenv, install my dependencies into them and then I zip my lambda deployment packages composed by:

1. The site-packages of the virtualenv
2. The sources

After that, I store my zip packages into S3 buckets and I update the code of the function with the aws cli. Everything seems to work fine but I have two problems:

First, the files seem far too light to me (3.8MB). And when I want to test my lambda functions it's like no modules have been installed. See the error below:

Unable to import module 'lambda_function': No module named 'gensim'

I think the virtualenv doesn't have the modules installed because when I downloaded the .zip files I could see that the __pycache__ folder only contains a small easy_install.cpython-36.pyc .

I don't know what I did wrong, but I think the problem comes from my deployment packages. Does anyone have any ideas?

Answer 1

Did you zip up the site-packages from both lib and lib64 in your virtualenv? I noticed that some package ended up in one place or other, and I have to package them up from both places.

See if you can find if gensim is installed inside site-packages in either lib/... or lib64/...

Answer 2

If you're deploying to AWS Lambda you're better off using a framework such as Serverless or Zappa to package your code and dependencies up into a zip file ready for deployment via S3.

Both work, though I prefer Serverless as it's pluggable, cross-language, seems to be better supported and Just Worked for me. This post gives a good rundown of how to get started.

Once you've got Serverless packaging working, your buildspec.yml becomes very simple: install serverless, run serverless package .

You can also use Serverless to manage the AWS infrastructure for you if you like. But I prefer to do that separately using Terraform .