Should PostgreSQL connections be pooled in a Python web app, or create a new connection per request?

Question

I\'m building a web app in Python (using Flask). I do not intend to use SQLAlchemy or similar ORM system, rather I\'m going to use Psycopg2 directly.

Should I open a new

Answer 1

I think connection pooling is the best thing to do if this application is to serve multiple clients and concurrently.

Answer 2

PgBouncer is pretty neat and transparent to the application and server.

We have been using PgBouncer in production for 2 years without a single issue. It's a pretty awesome PostgreSQL connection pooler.

http://wiki.postgresql.org/wiki/PgBouncer

Answer 3

The answer depends on how many such requests will happen and how many concurrently in your web app ? Connection pooling is usually a better idea if you expect your web app to be busy with 100s or even 1000s of user concurrently logged in. If you are only doing this as a side project and expect less than few hundred users, you can probably get away without pooling.

Answer 4

Pooling seems to be totally impossible in context of Flask, FastAPI and everything relying on wsgi/asgi dedicated servers with multiple workers. Reason for this behaviour is simple: you have no control about the pooling and master thread/process. A pooling instance is only usable for a single thread serving a set of clients - so for just one worker. Any other worker will get it's own pool and therefore there cannot be any sharing of established connections.

Logically it's also impossible, because you cannot share these object states across threads/processes in multi core env with python (2.x - 3.8).

Answer 5

Yeah connection pooling will help but yes you have to find out the right numbers for live connection or pool size based on which load will be on database.

    from psycopg2.pool import SimpleConnectionPool
    from contextlib import contextmanager

    dbConnection = "dbname='dbname' user='postgres' host='localhost' password='postgres'"

    # pool define with 10 live connections
    connectionpool = SimpleConnectionPool(1,10,dsn=dbConnection)

    @contextmanager
    def getcursor():
        con = connectionpool.getconn()
        try:
            yield con.cursor()
        finally:
            connectionpool.putconn(con)

    def main_work():
        try:
            # with here will take care of put connection when its done
            with getcursor() as cur:
                cur.execute("select * from \"TableName\"")
                result_set = cur.fetchall()

        except Exception as e:
            print "error in executing with exception: ", e