appengine datastore model for twitter like showing posts from users followed

Question

I am working on a web system where a feature is similar to Twitter's concept of following a list of users and seeing their posts as a list.

The simple model I came up with requires join operation which is not available in datastore .

class Post(Model):
   author = reference to user id
   content = text content

class Following(Model):
   author = reference to user id
   followed_by = reference to user id

The frequent operation is to display a list of posts (sorted in time) from users followed by the current user.

With the above model, it can only be done in two steps:

authors = Following.author when Following.followed_by == current_user
posts = Posts with Posts.author in authors

Is there any way to achieve this more efficiently?

Answer 1

You can use a structure property to store all Posts within the Author object.

There's an interesting discussion here that might be interesting for you to choose which approach is the best for your use case.

Answer 2

You could use a single query for displaying posts if you change the algorithm a bit. You could track what posts would need to be displayed for a particular user, with entities like these:

class DisplayPost(Model):
   #parent entity = user for which the post should be displayed
   #key ID matches the Post's key ID
   posted = datetime  # if you want timed ordering in display
   expiry = datetime  # optional for periodic cleanup jobs

Whenever an author creates a new post you'd just launch task(s) to create such entities for every follower of the author.

Whenever you need to display posts for a user you'd make a single ancestor keys_only query to get the list of DisplayPost keys:

keys = DisplayPost.query(ancestor=user_key, ...).fetch(keys_only=True)

From this you obtain a corresponding list of Post keys and get the post with a get_multi() op, something along these lines:

post_keys = [ndb.Key(Post, key.id()) for key in keys]
posts = ndb.get_multi(post_keys)

This allows you a much faster response time when displaying posts, with no join and no IN (also problematic) ops. Better scalability. The price to pay is always preparing the DisplayPost , even if some of them will never be used (if the respective users don't even log in, for example).