How to tune a cache in an application server

Question

Our customers have hundreds of projects per application server.

Every project has its own metadata descriptor (~1MB in memory , ~1sec to load from DB). The metadata is used to validate every request to the server in various aspects (permissions, valid fields, field values etc...) We are heavily dependent upon this metadata.

To save time querying the DB, we decided to implement cache mechanism (using EHCache) to store the project's metadata. And we would like to tune the cache properly.

I was wondering, what are the techniques for tuning the cache properties (total size, number of objects, evacuation strategies ...)

For example, should objects that haven't been accessed for a period of time be released to save memory? How to choose eviction strategy (eg LRU, MRU)?

We do have an experienced load testing team and lots of data from customers (number of projects, average metadata size, etc...) so building load testing environment shouldn't be a problem. I am just unsure what exactly should I look for?

Please share your cache tuning experience.

Thanks

Answer 1

The key stat when it comes to caches is pretty simple... cache miss rate. You want to count the number of times someone asked the cache for something, and the number of times that "something" was not already in the cache, and the latter divided by the former is your miss rate.

That's pretty much it. Find a reliable way to measure that for your application and everything else is simply running the tests. Every time you tune, load test under real-world circumstances to examine changes to the miss rate. The available parameters are far too dependent on the specifics of your problem.

I would generally start with LRU, increase size until I start seeing diminishing returns on the miss rate, and then try out LFU at that same size to see how it compares. Unless, of course, you happen to know for sure that your case is pathological to LRU.

Answer 2

The optimal eviction strategy may depend a lot on the specific data and how it's used.

Here are a couple of generic examples:

News Content - articles are added on a daily basis and rarely if ever modified - cache is used to reduce demand on the database server. Eviction strategy should be tailored around many reads but almost no writes.

Forum Threads (may not be the best example) - Eviction strategy should be tailored around many reads, but also around frequent writes.

While the news content may be most optimally configured with a Least Recently Used (LRU) eviction strategy, the forum threads in this example may be more optimally configured with a Least Frequently Used (LFU) eviction strategy. The cache for the forum thread would need to be frequently replaced anyway when new messages are added, however, an LFU strategy would keep the most active threads in cache.