I'm trying to create a new lambda layer to import the zip file with psycopg2, because the library made my deployment package get over 3MB, and I can not see the inline code in my lambda function any more.
I created lambda layer for the following 2 cases with Python 3.7:
I added they new created layer into my lambda function. But, in both cases, my lambda_function throws an error as follows: { "errorMessage": "Unable to import module 'lambda_function': No module named 'psycopg2'", "errorType": "Runtime.ImportModuleError" }
The error seems as if something went wrong with my zip file that they are not recognized. But when it works well in my deployment package.
Any help or reason would be much appriciated. Thanks!
not sure if the OP found a solution to this but in case others land here. I resolved this using the following steps:
mkdir -p python/lib/python3.7/site-packages/psycopg2
cp psycopg2-3.7/* python/lib/python3.7/site-packages/psycopg2
zip -r9 psycopg2-py37.zip python
If you end up on this page in >= 2022 year. Use official psycopg2-binary https://pypi.org/project/psycopg2-binary/
Works well for me. Just
pip install --target./python psycopg2-binary
zip -r python.zip python
Maintainers of psycopg2 do not recommend using psycopg2-binary
because it comes with linked libpq
and libssl
and others that may cause issues in production under certain circumstances.
I may imagine this being an issue when upgrading postgresql server while bundled libpq
is incompatible. I also had issues w/ psycopg2-binary
on AWS Lambda running in arm64 environment.
I've resorted to building postgresql
and psycopg
in Docker running on linux/arm64
platform using public.ecr.aws/lambda/python:3.9
as the base image.
FROM public.ecr.aws/lambda/python:3.9
RUN yum -y update && \
yum -y upgrade && \
yum -y install libffi-devel postgresql-devel postgresql-libs zip rsync wget openssl openssl-devel && \
yum -y groupinstall "development tools" && \
pip install pipenv
ENTRYPOINT ["/bin/bash"]
The build script is the following and valid for aarch64
platform. Just change path to x86_64 version on Prepare psycopg2
step.
#!/usr/bin/env bash
set -e
PG_VERSION="14.5"
cd "$TERRAFORM_ROOT"
if [ ! -f "postgresql-$PG_VERSION.tar.bz2" ]; then
wget "https://ftp.postgresql.org/pub/source/v$PG_VERSION/postgresql-$PG_VERSION.tar.bz2"
tar -xf "$(pwd)/postgresql-$PG_VERSION.tar.bz2"
fi
if [ ! -d "psycopg2" ]; then
git clone https://github.com/psycopg/psycopg2.git
fi
# Build postgres
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION"
./configure --without-readline --without-zlib
make
make install
# Build psycopg2
cd "$TERRAFORM_ROOT/psycopg2"
make clean
python setup.py build_ext \
--pg-config "$TERRAFORM_ROOT/postgresql-14.5/src/bin/pg_config/pg_config"
# Prepare psycopg2
cd build/lib.linux-aarch64-3.9
mkdir -p python/
cp -r psycopg2 python/
zip -9 -r "$BUNDLE" ./python
# Prepare libpq
cd "$TERRAFORM_ROOT/postgresql-$PG_VERSION/src/interfaces/libpq/"
mkdir -p lib/
cp libpq.so.5 lib/
zip -9 -r "$BUNDLE" ./lib
where $BUNDLE
is the path to already existing.zip file.
I also tried to statically build psycopg2 binary and link libpq.a
, however, I have had quite a lot of issues with missing symbols.
From AWS post How do I add Python packages with compiled binaries to my deployment package and make the package compatible with Lambda? :
To create a Lambda deployment package or layer that's compatible with Lambda Python runtimes when using pip outside of Linux operating system, run the pip install command with
manylinux2014
as the value for the--platform
parameter.
pip install \
--platform manylinux2014_x86_64 \
--target=my-lambda-function \
--implementation cp \
--python 3.9 \
--only-binary=:all: --upgrade \
psycopg2-binary
You can then zip the content of directory my-lambda-function
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.