Designing a global action counter in high traffic web development

Question

We have a webapp that caters to hundreds of simultaneously logged in users (about 10K-30K users at any given time). The app collects analytics, specifically on certain user actions that may occur a few times a second.

So far our app design has been pretty decoupled (a lot of memcache/redis with delayed DB writes) and we avoided locks pretty well to make sure nothing is "centralized".

Management finally decided to build a real time analytics panel that should aggregate these actions in global counters (down to 1 second granularity). Whats the best way to have these "global" counters? We could increment some memcache key but we have a cluster of memcaches (EC2) so iterating over all of them to count up the keys would delay this metric. DB is out of the question since we were bottlenecking alot in that regard so all DB writes are delayed thru a message queue (beanstalkd)

Any tips would be highly appreciated.

Answer 1

This would appear suited to a NoSQL dump of the actions, with periodic agregation. And being on EC2, you're in the right place to have access to the tools you need.

You could avoid your existing webserver infrastructure entirely by setting up a secondary webserver to record all the actions, pumping into a separate database server. Or if not appropriate, share the webserver but still offload to a separate NoSQL server.

Then, if "real time" can be delayed by a small period (seconds or a few minutes), you can have a sweeper function that agregates the NoSQL table into a format that more suits the analytics system, and pumps into your "live" database and clears out NoSQL data that has been processed.

Alternatively, you may be able to get your stats directly from the NoSQL?

NoSQL may be as fast as using Memcached (various benchmarks report various results, depending on who wrote the report) but it'll certainly be faster in pulling the data together when you need to agregate.