Google App Engine Efficient Data store Read/Write Operation to Save Quota

Question

I've created an Google App Engine Apps using Python.The application deals with lot of user names.

It has got a database to 50K usernames. Each user name has a unique hash value. Which is also stored in the data store.

When any app user submit any user name. The application first checks if the username exist in DB.

If its a new user name, the application calculate a new hash for the new name and store the name and hash in DataStore.

If user name already exist in Datastore, it retrieve the old hash from data store.

Sample Code:

class Names(db.Model):
    name = db.StringProperty(required=True)
    hash = db.StringProperty(required=True)

username = "debasish"
user_db = db.GqlQuery("SELECT * FROM Names WHERE name=:1", username)
user = user_db.get()
if user == None:
    #doesn't exist in DB..so calculate new hash for that name and store it in DB
    e = Names(name=username,hash="badasdbashdbhasbdasbdbjasbdjbasjdbasbdbasjdbjasbd")
    e.put()
else:
    #retrieve the old hash.
    self.response.out.write('{"name":"'+user.name+'","hash":"'+user.hash+'"}')            

The problem I'm facing is GAE's free data store read operation quota.Its exceeding too quickly and My application stop working.

I've also tried to implement memcache,like this , adding entire db in memcache. But it was also a failure,result more bad.

def get_fresh_all(self):
    all_names = db.GqlQuery("SELECT * FROM Names")
    memcache.add('full_db', all_names, 3600)
    return all_names

So,guys could you please suggest , Am I doing something wrong?? How I can make data store read operations more efficiently??

Thanks in Adv.

Answer 1

you can:

• switch to NDB where caching is automatic
• query the keys instead of entities SELECT __key__ FROM ...
• reduce the related indexes (surely decreases the write ops, perhaps even read ops)
• rewrite all your entities with username as key_name and use the method get_or_insert()

user = Names.get_or_insert("debasish", hash="badasdbashdbhasbd")

Answer 2

You should cache only the username = hash instead of all. Plus add a in memory cache (this works per instance only cache. Should help more, just create a dict on global module level). This could grow really quickly depending on your unique hits but you can add a logic to only hold certain numbers. Here is a sample:

cache = {}

def get_user_hash(username):
    if username in cache:
         return cache[username]
    hash = memcache.get(username)
    if not hash:
        hash = # retrieve from db
        if not hash:
            # put to db & assign hash=new_hash

        cache[username] = hash
        memcache.set(username, hash)
    return hash

Answer 3

@Faisal's method should work well, it adds two levels of caching to the query.

Another option is to store the username and hash in the session. Only check the database once per session, and then retrieve the values from the session variables.