Python async transactions psycopg2

Question

It is possible to do async i/o with psycopg2 (which can be read here ) however I'm not sure how to do async transactions. Consider this sequence of things:

• Green Thread 1 starts transaction T
• GT1 issues update
• GT2 issues one transactional update
• GT1 issues update
• GT1 commits transaction T

I assume that GT1 updates conflict with GT2 updates.

Now according to docs :

Cursors created from the same connection are not isolated, ie, any changes done to the database by a cursor are immediately visible by the other cursors.

so we can't implement the flow above on cursors. We could implement it on different connections but since we are doing async then spawning (potentially) thousands db connections might be bad (not to mention that Postgres can't handle so much out-of-the-box).

The other option is to have a pool of connections and reuse them. But then if we issue X parallel transactions all other green threads are blocked until some connection is available. Thus the actual amount of useful green threads is ~X (assuming the app is heavily db bound) which raises question: why would we use async to begin with?

Now this question can actually be generalized to DB API 2.0. Maybe the real answer is that DB API 2.0 is not suited for async programming? How would we do async io on Postgresql then? Maybe some other library?

Or maybe is that because the postgresql protocol is actually synchronous? It would be perfect to be able to "write" to any transaction at any time (per connection). Postgresql would have to expose transaction's id for that. Is it doable? Maybe two-phase commit is the answer?

Or am I missing something here?

EDIT: This seems to be a general problem with SQL since BEGIN; COMMIT; BEGIN; COMMIT; semantics just can't be used asynchronously efficiently.

Answer 1

Actually you can use BEGIN; and COMMIT; with async. What you need is a connection pool setup and make sure each green thread gets its own connection (Just like a real thread would in a multithreaded application).

You cannot use psycopg2's builtin transaction handling.