MWAA Airflow job getting SCRAM error when connecting to postgres

Question

I'm trying to query postgres from an MWAA instance of airflow. I'm not sure if there is a conflict due to airflow itself having a different version of postgres for its metadata or what, but I get this error when connecting to postgres:

  File "/usr/local/airflow/dags/transactions/transactions.py", line 62, in load_ss_exposures_to_s3
    ss_conn = psycopg2.connect(
  File "/usr/local/airflow/.local/lib/python3.10/site-packages/psycopg2/__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
psycopg2.OperationalError: SCRAM authentication requires libpq version 10 or above

Locally I have psycopg2 version 2.9.5 and libpq version 140005. MWAA is using psycopg2 2.9.5 and libpq 90224. Is there a way for me to force MWAA to use another version? Maybe through airflow plugins? Airflow version is 2.4.3.

Answer 1

To update the libs version, in AWS MWAA, you can add a requirements file to S3, and configure MWAA to install it by selecting the file and updating the environment ( doc ).

For your problem, you can try installing the same version you have in localhost, but if you are using graviton VM in MWAA (arm64), it can be another problem, like installing psycopg2-binary which is not compatible with arm64, and in this case you need to install psycopg2 :

pip install psycopg2