No module named 'psycopg2._psycopg': ModuleNotFoundError in AWS Lambda

Question

I have created a deployment package for AWS Lambda with my python file and the dependencies including sqlalchemy and psycopg2. The code works perfectly in accessing the DB locally. But when I imported this zip file, I am getting the following error.

No module named 'psycopg2._psycopg': ModuleNotFoundError

The stack trace of the error is,

{
  "errorMessage": "No module named 'psycopg2._psycopg'",
  "errorType": "ModuleNotFoundError",
  "stackTrace": [
    [
      "/var/task/DBAccessLamdaHandler.py",
      50,
      "lambda_handler",
      "engine = create_engine(rds_host)"
    ],
    [
      "/var/task/sqlalchemy/engine/__init__.py",
      387,
      "create_engine",
      "return strategy.create(*args, **kwargs)"
    ],
    [
      "/var/task/sqlalchemy/engine/strategies.py",
      80,
      "create",
      "dbapi = dialect_cls.dbapi(**dbapi_args)"
    ],
    [
      "/var/task/sqlalchemy/dialects/postgresql/psycopg2.py",
      554,
      "dbapi",
      "import psycopg2"
    ],
    [
      "/var/task/psycopg2/__init__.py",
      50,
      "<module>",
      "from psycopg2._psycopg import (                     # noqa"
    ]
  ]
}

Any help is appreciable

Answer 1

The AWS Lambda runtime environment doesn't include the PostgreSQL libraries so you need to include them within your AWS Lambda upload.

One way to do this is to get them from the jkehler/awslambda-psycopg2 repo at GitHub. Note that you don't need to build this project from scratch as the repo includes a pre-built package in the psycopg2 folder that you can simply include in your Lambda upload.

Answer 2

The psycopg2 build library from jkehler/awslambda-psycopg2 was built for python 3.6 and make sure that while uploading your code to AWS lambda, select Python Runtime environment as 3.6, and it should work. I banged my head on this for a full day and then when i changed to 3.6, the import error just vanished.

If you are going to attempt to build it yourself, remember that you must build on a machine or VM with the same architecture as your target at AWS .

Answer 3

Latest as of 26/MAR/2020

I was skeptical about depending on a third-party library for my production code. On research the following works,

The issue happens only when the packages are built from MAC OS.

I can confirm today that the issue is fixed when I build the package from Centos 7 ( AWS AMI )

The following is my approach

requirement.txt

psycopg2-binary==2.8.4

Build process

pip install -r requirements.txt --target .

Lambda code is in the root directory

+-- lambda_function.py
+-- psycopg2
    +-- psycopg2 files

Zip the directory and test the code in lambda works.

The only additional step is to build the package in Linux env instead of macOS using the Docker container. An example can be found here: Deploy AWS Amplify Python Lambda from macOS with Docker

Answer 4

For Windows (Looks like in any OS which is not build on Amazon AMI or Centos), the easiest fix is to use

psycopg2 Python Library for AWS Lambda

which you can find https://github.com/jkehler/awslambda-psycopg2

Answer 5

Like other answers, psycopg2-binary worked fine for python3.9 (it looks like other package awslambda-psycopg2 was only available for python3.6).

But, if you run on MacOs before sending to aws lambda, you must specify the platform to your pip install like this:

pip3.9 install --platform=manylinux1_x86_64 --only-binary=:all: psycopg2-binary

Answer 6

instead of using psycopg2, try using pg8000 in command prompt go the directory that you are currently working(example F:\example) pip install pg8000 -t. (-t. is for installing pg8000 in the directory that you are working)

// handler.py import pg8000

database=''
host=''
port=''
user=''
password=''
conn = pg8000.connect(database=database, host=host, port=port, user=user, 
password=password)

def lambda_function(event, context):
   .
   .
   .

after writing your code zip the code and upload in your aws lambda.

It worked for me!!!

Answer 7

I've had issues getting SQLAlchemy to run on AWS Lambda despite the fact that I have tried multiple psycopg2 versions, but what eventually solved it for me was to use an older Python version. I went from Python 3.9 to 3.7 and it was finally able to run (Using psycopg2-binary 2.8.4, but I didn't try other versions or the none binary version with 3.7)

Answer 8

Thanks to AWS Lambda Layers we can include a ready compiled psycopg2 layer for our selected Python versions directly to our Lambda function. Please use the version you need from this Github repo . Be careful to use the correct Python version and Region in Lambda and layer creation.

Answer 9

I have faced a similar issue. I tried to create the layers using AWS Cloud shell it worked with Python3.8 . But failed in Python3.9 though.

Please change the region name as per your AWS Region

sudo amazon-linux-extras install python3.8

curl -O https://bootstrap.pypa.io/get-pip.py

python3.8 get-pip.py --user

mkdir python

python3.8 -m pip install --platform=manylinux1_x86_64 --only-binary=:all: pandas numpy pymysql psycopg2-binary SQLAlchemy -t python/

zip -r layer.zip python

aws lambda publish-layer-version \
 --layer-name mics-layer \
 --description "Pandas Numpy psycopg2-binary SQLAlchemy pymysql" \
 --zip-file fileb://layer.zip \
 --compatible-runtimes python3.8 python3.9 \
 --region eu-west-1

Answer 10

In CDK you can use something like this. Take in consideration that this only works in Python 3.8 .

    sqlachemy_layer = lambda_.LayerVersion(
        scope=self,
        id=LAYERS_SQLALCHEMY_LAYER,
        layer_version_name=LAYERS_SQLALCHEMY_LAYER,
        code=lambda_.Code.from_asset(
            str(pathlib.Path(__file__).parent.joinpath("dependencies").resolve()),
            bundling=BundlingOptions(
                image=lambda_.Runtime.PYTHON_3_8.bundling_image,
                command=[
                    "bash", "-c",
                    "pip install -r requirements-sqlalchemy.txt -t /asset-output/python && cp -au . /asset-output/python",
                ]
            )
        ),
        compatible_runtimes=[
            lambda_.Runtime.PYTHON_3_8,
        ],
    )

The requirements-sqlalchemy.txt:

psycopg2-binary
sqlalchemy

And the file structure:

• dependencies/
  • requirements-sqlalchemy.txt
• my_stack.py

Answer 11

The latest update to psycopg (when I answered this), fixes this. You need to do update the version for psycopg2-binary to 2.8.4. That did the trick for me.

psycopg2-binary==2.8.4