如何在1秒内从GAE数据存储中检索大量（> 2000）数量的实体？

Question

We have some part of our application that need to load a large set of data (>2000 entities) and perform computation on this set. The size of each entity is approximately 5 KB.

On our initial, naïve, implementation, the bottleneck seems to be the time required to load all the entities ( ~40 seconds for 2000 entities ), while the time required to perform the computation itself is very small (<1 second).

We had tried several strategies to speed up the entities retrieval:

• Splitting the retrieval request into several parallel instances and then merging the result: ~20 seconds for 2000 entities .
• Storing the entities at an in-memory cache placed on a resident backend: ~5 seconds for 2000 entities .

The computation needs to be dynamically computed, so doing a precomputation at write time and storing the result does not work in our case.

We are hoping to be able to retrieve ~2000 entities in just under one second. Is this within the capability of GAE/J? Any other strategies that we might be able to implement for this kind of retrieval?

UPDATE: Supplying additional information about our use case and parallelization result:

• We have more than 200.000 entities of the same kind in the datastore and the operation is retrieval-only.
• We experimented with 10 parallel worker instances, and a typical result that we obtained could be seen in this pastebin . It seems that the serialization and deserialization required when transferring the entities back to the master instance hampers the performance.

UPDATE 2: Giving an example of what we are trying to do:

1. Let's say that we have a StockDerivative entity that need to be analyzed to know whether it's a good investment or not.
2. The analysis performed requires complex computations based on many factors both external (eg user's preference, market condition) and internal (ie from the entity's properties), and would output a single "investment score" value.
3. The user could request the derivatives to be sorted based on its investment score and ask to be presented with N-number of highest-scored derivatives.

Answer 1

200.000 by 5kb is 1GB. You could keep all this in memory on the largest backend instance or have multiple instances. This would be the fastest solution - nothing beats memory.

Do you need the whole 5kb of each entity for computation? Do you need all 200k entities when querying before computation? Do queries touch all entities?

Also, check out BigQuery . It might suit your needs.

Answer 2

Use Memcache . I cannot guarantee that it will be sufficient, but if it isn't you probably have to move to another platform.

Answer 3

In the end, it does not appear that we could retrieve >2000 entities from a single instance in under one second, so we are forced to use in-memory caching placed on our backend instance, as described in the original question. If someone comes up with a better answer, or if we found a better strategy/implementation for this problem, I would change or update the accepted answer.

Answer 4

Our solution involves periodically reading entities in a background task and storing the result in a json blob. That way we can quickly return more than 100k rows. All filtering and sorting is done in javascript using SlickGrid's DataView model.

As someone has already commented, MapReduce is the way to go on GAE. Unfortunately the Java library for MapReduce is broken for me so we're using non optimal task to do all the reading but we're planning to get MapReduce going in the near future (and/or the Pipeline API).

Mind that, last time I checked, the Blobstore wasn't returning gzipped entities > 1MB so at the moment we're loading the content from a compressed entity and expanding it into memory, that way the final payload gets gzipped. I don't like that, it introduces latency, I hope they fix issues with GZIP soon!

Answer 5

This is very interesting, but yes, its possible & Iv seen some mind boggling results.

I would have done the same; map-reduce concept

It would be great if you would provide us more metrics on how many parallel instances do you use & what are the results of each instance?

Also, our process includes retrieval alone or retrieval & storing ?

How many elements do you have in your data store? 4000? 10000? Reason is because you could cache it up from the previous request.

regards