Some properties of an Entity need full-text search, others don't

Question

In my app I have an entity called Product that holds the details of the product listings to be shown on the site. Since the name (title) and description of the product need to be full-text searchable, I am putting them in a separate Document (as defined by Google's App Engine Search API ). All other properties remain in the Product entity, which is stored as per usual in the Google Datastore using Objectify .

A slug created from the name property of the Product is the doc_id of the Document . This same slug is the ( String ) ID of an entity (let's call it ProductLookup ) that I will use to get the Key to the corresponding/matching Product .

NOTE: The Product has an auto-generated Long id as the name property (and therefore the slug used to lookup a Product ) can change even after the creation of the Product — this way I just create a new ProductLookup entity when name changes.

Since I am splitting the information that would normally be in a single entity into two different objects (the Product and the Document , not to mention the extra ProductLookup entity), are there any special cases I should be watching out for? Any suggestions?

---

If I have a reference to a Product , I can use the slug to get the correspoding Document .

If I have a slug, I can use ProductLookup to get a Product and use the slug as the doc_id of the Document .

If I find a set of Document s via Search API, I can use their doc_id s to get the ProductLookup , then the Product .

Answer 1

My current project has a similar situation. Originally I was using a technique like yours, and the only caveat I know of is the extra care you have to take to ensure adding and deleting products from the two sources always succeed or fail together. Datastore operations can be put in a transaction, but Search API operations will not obey the rules of that transaction. For example, if you delete from both the datastore and the search api within a transaction, and that transaction fails, the product will not be deleted from the datastore, but it may still be deleted from the search api. To ensure they stay synchronized, use a technique like this , and be sure your search api operations are last within your transactions.

I have since amended my approach to save the entire object within the datastore, and also store a copy of the searchable fields in the search api. The drawback to this approach is that you pay for the additional storage. The benefit is that when you aren't searching, retrievals are cheaper. This is especially true when Objectify finds your objects in memcache.

To support my approach, I wrote my own "mini-objectify" that persists objects to the search API. It has its own version of @Igore , so I can use the same POJO for both, and only persist the fields I choose to the search api.