Google App Engine: How to do queries interact when they execute a function decorated by ndb.transaction

Question

Just to understand the inner workings of ndb.transaction, I tried the following experiment.

The Conference API in Udacity example was used to create a conference with just one seat available.

I added a wait in the method that handles the registerForConference API as shown below. I put logging debug messages to understand the flow as shown below

I started two calls of the registerForConference API one after the other.

Without ndb.transaction, both will return true (ie both return registed successfully). With ndb.transaction, the second will return false. So, things are working as expected.

But the sequence of the steps the second request went through was unexpected. I was expecting the second query to get stuck at some point until the first one completes and a TransactionFailedError will be thrown when it tries to insert. Instead, it looks like the second request actually falls through the method once and then re-executes the method after the first one completes. In the second exception it reads the updated value of seatsAvailable and determines it can't register. Is this the expected behavior? If yes, isn't this wasteful since some of the steps can be done in parallel and only steps with conflict need to be executed after first query completes?

Sequence of debug log messages printed

/_ah/spi/ConferenceApi.regForAConf
D 18:28:21.093 Checking for id_token.
D 18:28:21.093 id_token verification failed: Token is not an id_token (Wrong number of segments)
D 18:28:21.093 Checking for oauth token.
D 18:28:21.101 Returning user from matched oauth_user.
D 18:28:21.111 Entered conf registration check
D 18:28:21.125 Got a profile object
D 18:28:21.131 Got a conf object
D 18:28:21.131 Entered updating step
**Went through the entire method once**
**Then restarted after first API completed**
D 18:28:46.143 Leaving with exit value 1
D 18:28:46.168 Entered conf registration check
D 18:28:46.181 Got a profile object
D 18:28:46.187 Got a conf object
D 18:28:46.187 Transaction failed No seats available
D 18:28:46.187 Leaving with exit value 0

Definition of method handling the API request

   @ndb.transactional(xg=True) 
def _conferenceRegistration(self,confId):
    #get conference from id
    ret_val =True
    user = endpoints.get_current_user()
    if not user:
        raise endpoints.UnauthorizedException('Authorization required')
    user_id = getUserId(user)
    logging.debug('Entered conf registration check')
    p_key = ndb.Key(Profile, user_id)
    prof = p_key.get()

    logging.debug('Got a profile object')
    conf_key = ndb.Key(urlsafe=confId)
    conf = conf_key.get()
    logging.debug('Got a conf object')

    if conf and prof:
        if conf.seatsAvailable>0:
            logging.debug('Entered updating step')
            conf.seatsAvailable=conf.seatsAvailable-1
            time.sleep(25)
            prof.conferencesToAttend.append(conf.name)
            try:
                conf.put() 
            except TransactionFailedError:
                logging.debug('Transaction Failed error when trying to insert changes to conference')
                ret_val=False

            try:
                prof.put() 
            except TransactionFailedError:
                logging.debug('Transaction Failed error when trying to insert changes to profile')
                ret_val=False

            ret_val=True
        else:
            logging.debug('Transaction failed No seats available')
            ret_val=False  
    else:
        logging.debug('Could not get conf or profile instance')
        ret_val=False

    buf =  'Leaving with exit value %d' % (ret_val)   
    logging.debug(buf)
    return BooleanMessage(regSucc=ret_val)

Answer 1

This is expected. It isn't always the most efficient method for handling transactions, but it's the model they've chosen to use - it assumes that transaction collisions will be rare, and just verifies that before writing. This is known as 'optimistic concurrency'. The alternative woluld involve locking things, which can be quite complicated, and less efficient when transactions don't often collide.

The documentation for how transactions work on appengine might help explain more, or there's a wikkipedia page on optimistic concurrency control