Handling transaction conflict with SQLAlchemy

Question

I'm relatively new to databases. I'm sure this is the sort of question that experience would answer.

I'm using SQLAlchemy with PostgreSQL. I have a system set up where multiple processes spread across several computers carry out various tasks, and then update the database. I haven't encountered any transaction conflicts yet in testing, but they're still theoretically possible.

From what information I've been able to find on Google, it looks like I'll either have to acquire a lock on the database, or be prepared to restart transactions. Unfortunately, there is precious little information on how to actually do this.

I assume that, to restart the transaction, there is some exception thrown by SQLAlchemy that my code will have to catch, and perform the retry itself. Is that exception distinct from the one SQLA would throw if I, say, violated a uniqueness constraint, indicating a bug in my code as opposed to a transaction conflict? Would I just be better off using a database lock instead?

Thanks in advance!

-- Edit --

I have just learned about "ConcurrentModificationError". The name sure sounds like the exeception I'm looking for. The documentation says it's the alias of StaleDataError , whose name again sounds right, but whose documentation is pretty opaque. Is this the error I'm looking for?

Again, thanks so much!

Answer 1

I have not ever seen that error, although the details for StaleModificationError suggest it might be the thing you are concerned about, but there should be no need to lock the entire database. You might read up on transaction isolation in Postgres to prevent the different process workers from unknowingly reading a row being updated in another transaction.

If you set your isolation level higher (read-committed, etc) then your SA session will start collecting locks on the various rows it is touching. You decide for your design how stringent to set the isolation level. You can make it strict enough s your readers will throw an exception when they try to read a row another transaction has locked, and then you can choose to roll back or use the refresh/expire interface to update that particular session.

Answer 2

Just adding a few details to khoxsey's answer.

Setting the transaction isolation level to "serializable" will make PostgreSQL raise an error if concurrent transactions interfere with one another. The "serializable" isolation level decreases performance when there are many concurrent database connections, but there are few enough in my case that performance isn't an issue.

When transactions conflict with one another at this serialization level, Postgres calls it a "serialization error", which has an SQLSTATE error code of "40001". There is currently no driver-portable way to access the SQLSTATE in SQLAlchemy. However, when using the psycopg2 driver, the SQLSTATE code is accessible through exc.orig.pgcode , where exc is the exception caught by an except statement.

From what I can tell, the exception itself should be an OperationalError.