How to store a huge (Oracle) dataset in memory cache

Question

Description: I have a transaction table, with average of ~200 (+-100) rows inserts per second, from which the present system queries to retrieve the data based on client ids with/without the time (order-by ascending) and few other conditions. This table is partitioned in such a way that the table contains data for current day only, the earlier data are stored on < table_name_date_year >. The query is run on this table to generate various detailed reports for the customers.

A recent change to improve performance, 1) fetch min and max of the table, 2) query the table on every 50000 records in parallel (only 5 at a time), run the query with conditions between the < from > and < to >, till the max is reached 3) generate report, 4) combine the reports.

The Problem: Having done various query tuning by the database developers, the query still runs slower (takes many minutes to finish).

Can an in-memory data cache like memcached or redis (or ehcache?) or "any other cache frameworks" be used to store/update the data stored against the client-id, such that the reporting system fetches the data from the cache and generates the reports, instead to query straight from thge primary-db? (example: < key, pair >, < client-id, { dataset } >) If so how?

Can a bigdata implementation like HAZELCAST or Terracotta or (I haven't explored much on this) or Apache Gora be used here? If so how?

Can a Java based solution be developed using MappedByteBuffer , but will this be efficient?

Answer 1

NoSQL databases can help you if you are willing to give up on ACID. Generally speaking you don't have the ability to do SQL queries with those, so you are going to deal with big key-value sets and lists. Don't go overboard with complex setups, try a naive approach with a POJO and see if it works, if not, try Redis.

Answer 2

The shortest path for you to explore is Oracle's TimesTen. Since it is also an Oracle product, it's going to (probably) be the easiest to integrate with your current system. At a (very) high level, you can just bolt it on and, poof, it's an in-memory cache for your Oracle database, and will automatically keep it in sync. And, you'll be able to use the same SQL operations.

If that doesn't get you to the performance you need, you could look at my company's product, eXtremeDB. We have quite a few customers that use it as a front-end cache to Oracle. It's not an Oracle product, so the SQL you use with Oracle will probably have to be tweaked, and there will be some work to do to keep eXtremeDB in sync with Oracle (but a whole lot less work than going with a NoSQL/key-value pair solution). The payoff is that eXtremeDB is a lot faster than TimesTen (Oracle's license doesn't permit publication of benchmarks, so I can't be more specific).