Flask: PK Violations Using flask-sqlalchemy

Question

I'm having concurrency problems leading to PK violations in flask-sqlalchemy with MySQL using uwsgi with two worker threads and nginx.

I'm using keys that come from the payload. If the contact comes in with a key that exists in the database, update the record, if not, create a new record.

Here is the setup and what I think is happening.

#project/__init__.py
app = Flask(__name__)
db = SQLAlchemy(app)
from project import views, models

#project/views.py
from project import db

@app.route('/receive', methods = ['POST'])
def receive():

    #Check to see if the contact exists in the database
    contact = models.Contact.getIssue(contact_payload['id'])

    if contact is None:
        #If not in the DB, create a new contact
        new_contact = models.Contact(contact_payload)
        db.session.merge(new_contact)
    else:
        #If in the DB, create an updated representation of this contact
        updated_issue = contact.updateContact(contact_payload)
        db.session.merge(updated_issue)

    ...Some other stuff...

    #Commit to DB
    db.session.commit()

    #Respond
    resp = jsonify({'Result' : 'Success'})
    resp.status_code = 200
    return resp

The issue comes when we receive two requests for the same contact at the exact same time (requestA is 12:10:49,063 and requestB is 12:10:49,066). One of the requests ends in a PK violation.

I suspect they're coming in on different worker threads and each request gets a scoped session (sessionA - requestA and sessionB - requestB) from flask-sqlalchemy.

I suspect both sessions contain nothing at the beginning of the requests that are now in the above code flow simultaneously.

requestA goes through the appropriate code flow and merges new_contact into sessionA.

At the same time requestB goes through the same code path where contact is None (because sessionA hasn't committed yet) and merges new_contact into sessionB.

Then sessionA commits before sessionB. When sessionB goes to commit, it gets the PK violation.

Can we do anything else other than catch the PK violation and react accordingly?

Answer 1

You have two options:

1. Catch the PK violation and react accordingly, like you already said.

2. Lock your transaction based on your id: this is more complicated, you need something to synchronize your locks, like redis. Take a look at python-redis-lock. It is just one option, the solution here is to avoid concurrency for a PK.

https://pypi.python.org/pypi/python-redis-lock