Apple M1： install psycopg2 package Symbol not found: _PQbackendPID

Question

pg_config

venv ❯ which pg_config

/Applications/Postgres.app/Contents/Versions/latest/bin/pg_config

pip3 install -r requirements.txt

requirements.txt

venv ❯ cat requirements.txt
-i https://mirrors.aliyun.com/pypi/simple/
alembic==1.4.2
amqp==2.6.0
billiard==3.6.3.0
celery==4.4.6
certifi==2020.6.20
chardet==3.0.4
click==7.1.2
emoji==0.5.4
fastapi==0.59.0
future==0.18.2
h11==0.9.0
httptools==0.1.1 ; sys_platform != 'win32' and sys_platform != 'cygwin' and platform_python_implementation != 'PyPy'
idna==2.10
kombu==4.6.11
mako==1.1.3
markupsafe==1.1.1
psycopg2-binary==2.8.5
pydantic==1.6
python-dateutil==2.8.1
python-editor==1.0.4
pytz==2020.1
redis==3.5.3
requests==2.24.0
six==1.15.0
sqlalchemy==1.3.18
starlette==0.13.4
tenacity==6.2.0
urllib3==1.25.9
uvicorn==0.11.5
uvloop==0.14.0 ; sys_platform != 'win32' and sys_platform != 'cygwin' and platform_python_implementation != 'PyPy'
vine==1.3.0
websockets==8.1

full error message

Process SpawnProcess-1:
Traceback (most recent call last):
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/subprocess.py", line 62, in subprocess_started
    target(sockets=sockets)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/main.py", line 382, in run
    loop.run_until_complete(self.serve(sockets=sockets))
  File "uvloop/loop.pyx", line 1456, in uvloop.loop.Loop.run_until_complete
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/main.py", line 389, in serve
    config.load()
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/config.py", line 288, in load
    self.loaded_app = import_from_string(self.app)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/importer.py", line 23, in import_from_string
    raise exc from None
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/uvicorn/importer.py", line 20, in import_from_string
    module = importlib.import_module(module_str)
  File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "./main.py", line 4, in <module>
    from api.api_v1.api import api_router
  File "./api/api_v1/api.py", line 4, in <module>
    from api.api_v1.endpoints import status
  File "./api/api_v1/endpoints/status.py", line 10, in <module>
    from db.session import Session
  File "./db/session.py", line 7, in <module>
    engine = create_engine(config.SQLALCHEMY_DATABASE_URI, pool_pre_ping=True)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/sqlalchemy/engine/__init__.py", line 500, in create_engine
    return strategy.create(*args, **kwargs)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/sqlalchemy/engine/strategies.py", line 87, in create
    dbapi = dialect_cls.dbapi(**dbapi_args)
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 753, in dbapi
    import psycopg2
  File "/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/lidashuang/Projects/starservices/venv/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so

Answer 1

Try installing postgres via homebrew:

brew install postgresql

For some reason, this seems to install the right libraries in the right directories so that you can properly build and import psycopg2-binary on an M1 Mac (worked in my case anyway).

After installing postgres with brew, uninstall then reinstall psycopg2, eg:

pip3.9 install psycopg2-binary --force-reinstall --no-cache-dir

Note that even though this will install a new version of postgres, you don't need to actually use the new version to make the psycopg2 import work.

Answer 2

Miniconda (an alternative to virtualenv) has a Python 3.8 installer that works with psycopg2 on M1 Macs without the _PQbackendPID error: https://docs.conda.io/en/latest/miniconda.html

Answer 3

As for people using poetry. After you installed postgres via homebrew, (Also Make sure the installation is successful)

brew install postgresql

Then I follow this steps to get it to work.

First I remove my current poetry env with

poetry env remove [your env]

Then, clear poetry cache

poetry cache clear --all [your cache]

eg

poetry cache clear --all .

Then spawn a new poetry env, and you can install psycopg2 with

poetry add psycopg2-binary

Answer 4

In my case, using the following worked:

1. using miniconda
2. using psycopg2-binary

Answer 5

brew install libpq
brew install openssl

Follow the instructions after brew completes both times.

Answer 6

Are you creating a virtual environment using pycharm?

Remove the virtual env (if already created)

Press ⌘, to open the project Settings/Preferences and go to Project | Python Interpreter. Then click The Configure project interpreter icon and select Show All. Then click on '-' to remove the venv.

Create the virtual environment manually using the below command

python3 -m venv /path/to/venv

Activate the virtual venv

source venv/bin/activate

Install binary package of psycopg2

pip3 install psycopg2-binary 

Answer 7

pip install psycopg2-binary --force-reinstall --no-cache-dir

Answer 8

Try writing to the terminal PyCharm

pip3 install psycopg2-binary --force-reinstall --no-cache-dir