App Engine Memcache a single value shared across all instances

Question

I need to share a single MemCache key in Google App Engine across all the instances. This key is used for reading and incrementing, in a high rate (500 / sec).

Google suggests not to use "hot keys" and they recommend using datastore instead. Since I cannot increment or decrement a value by a single command in datastore, this is not usable for me.

What other option do I have?

Thanks

Answer 1

What you're looking for is called Sharding Counters.

Basically you need a few CounterShard entities to store value. You break your increment step into independent counters, then load and sum all values when you need it (it's cheap).

For 500/sec I guess you need up to 100 shards, but I suggest you to make few experiments to find best value.

There're full example hot to use it: https://cloud.google.com/appengine/articles/sharding_counters