Gevent with Database connection pool for Stream Server

Question

I am using Python Gevent's stream servers for communicating with another machine (remote) which sends concurrent TCP/IP requests (on an avg 60 req/sec). Nature of this communication is mostly IO bound (short text and then audio streams). I intend to use Postgresql to store the result of each communication (eg: filename received from the remote server).

I think its bad idea to invoke a new db connection for every greenlet spawned in Streamserver (pool size 90 so 90 req/sec max, that is the max I expect and avg 60 req/sec). Is it possible to have db connection pool which can be queued and every greenlet when starts running the handler function get a db connection from pool? Is there any tutorial that has worked for production systems? How would you suggest? I am using gevent 0.13.8 and postgres 9.1 with Python 2.7.3 on Ubuntu 10.04 64bit.

Answer 1

Gevent includes a postgres database pool in the examples:

https://github.com/gevent/gevent/blob/master/examples/psycopg2_pool.py

Answer 2

An alternative to in-application pooling is to pool using PgBouncer . You still have the overhead of a TCP connection and a little bit of setup, but massively less then making a full new PostgreSQL session.

Unlike in-application pooling, PgBouncer can be introduced transparently into existing systems as an intermediary between PostgreSQL and the pool. Just move PostgreSQl to port 5433 and have PgBouncer listen on port 5432.

Answer 3

I actually answered a similar answer here:
Python Postgres psycopg2 ThreadedConnectionPool exhausted

Basically I am setting up an async connection pool with gevent, starting up a Threadpool connection via postgres, and adding the "green" connection state to postgres.

So basically this creates a preset pool of connections that you specify (like 100) and then queues queries using this pool, as one query comes back a new query is handled. I like this approach better than a normal postgres pool because of the async nature of the requests, and the ease to implement inside a web framework.