如何使用 Flask-SQLAlchemy 避免 QueuePool 限制错误？

Question

I'm developing a webapp using Flask-SQLAlchemy and a Postgre DB, then I have this dropdown list in my webpage which is populated from a select to the DB, after selecting different values for a couple of times I get the "sqlalchemy.exc.TimeoutError:".

My package's versions are:

Flask-SQLAlchemy==2.5.1
psycopg2-binary==2.8.6
SQLAlchemy==1.4.15

My parameters for the DB connection are set as:

app.config['SQLALCHEMY_POOL_SIZE'] = 20
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 20
app.config['SQLALCHEMY_POOL_TIMEOUT'] = 5
app.config['SQLALCHEMY_POOL_RECYCLE'] = 10

The error I'm getting is:

sqlalchemy.exc.TimeoutError: QueuePool limit of size 20 overflow 20 reached, connection timed out, timeout 5.00 (Background on this error at: https://sqlalche.me/e/14/3o7r)

After changing the value of the 'SQLALCHEMY_MAX_OVERFLOW' from 20 to 100 I get the following error after some value changes on the dropdown list.

psycopg2.OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL:  sorry, too many clients already

Every time a new value is selected from the dropdown list, four queries are triggered to the database and they are used to populate four corresponding tables in my HTML with the results from that query.

I have a 'db.session.commit()' statement after every single query to the DB, but even though I have it, I get this error after a few value changes to my dropdown list.

I know that I should be looking to correctly manage my connection sessions, but I'm strugling with this. I thought about setting the pool timeout to 5s, instead of the default 30s in hopes that the session would be closed and returned to the pool in a faster way, but it seems it didn't help.

As a suggestion from @snakecharmerb, I checked the output of:

select * from pg_stat_activity;

I ran the webapp for 10 different values before it showed me an error, which means all the 20+20 sessions where used and are left in an 'idle in transaction' state.

Do anybody have any idea suggestion on what should I change or look for?

Answer 1

You are leaking connections.

A little counter intuitively, you may find you obtain better results with a lower pool limit. A given python thread only needs a single pooled connection, for the simple single-database queries you're doing. Setting limit to 1 , with 0 overflow, will cause you to notice a leaked connection earlier. This makes it easier to pin blame on the source code that leaked it. As it stands, you have lots of code and the error is deferred until after a bunch of queries have been issued, making it harder to reason about system behavior. I will assume you're using sqlalchemy 1.4.29.

To avoid leaking, try using this:

from contextlib import closing
from sqlalchemy import create_engine, text
from sqlalchemy.orm import scoped_session, sessionmaker

engine = create_engine(some_url, future=True, pool_size=1, max_overflow=0)
get_session = scoped_session(sessionmaker(bind=engine))
...
with closing(get_session()) as session:
    try:
        sql = """yada yada"""
        rows = session.execute(text(sql)).fetchall()
        session.commit()
        ...
        # Do stuff with result rows.
        ...
    except Exception:
        session.rollback()

Answer 2

I found a solution to the issue I was facing, in another post from StackOverFlow.

When you assign your flask app to your db variable, on top of indicating which Flask app it should use, you can also pass on session options, as below:

from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy(app, session_options={'autocommit': True})

The usage of 'autocommit' solved my issue.

Now, as suggested, I'm using:

app.config['SQLALCHEMY_POOL_SIZE'] = 1
app.config['SQLALCHEMY_MAX_OVERFLOW'] = 0

Now everything is working as it should.

The original post which helped me is: Autocommit in Flask-SQLAlchemy

@snakecharmerb, @jorzel, @J_H -> Thanks for the help!