How can I import pandas on AWS lambda?

Question

I got an error while I was trying to import pandas. I created a layer by uploading my package pandas zipped (from windows). I think it's because I zipped my package on windows I don't have the correct structure (same structure as I could get on Linux if zip a file). I read a lot of tutorials, but there are all based on mac or Linux, someone has an idea, to fix this issue?

Answer 1

I have done work like this. Launch an EC2 with AWS Linux2 then using python3 you create a venv and pip pandas.

python3 -m venv /path/to/new/virtual/environment https://docs.python.org/3/library/venv.html

If pandas and it dependancies are native, it will be created. Which is probably why the zip created on the windows box failed. My fail was I created the venv with dependancies on OSX, also not usable on AWS Linux:-D

You might need to install the development bundle.

yum groupinstall "Development tools"

Follow the instructions here to create the zip file: https://docs.aws.amazon.com/lambda/latest/dg/lambda-python-how-to-create-deployment-package.html

Take the zip file and save it to S3.

As you update the main lambda function, pull the zip from S3 and update your zip and push to lambda

If new packages are needed, recreate the base zip that is stored on S3.

I wrote code to do all this for work. Hence I cannot just provide the python I wrote that deals with all this. Since, work.

If you you want to build and test this in docker, use this image: https://github.com/lambci/docker-lambda