创建聚合表的最佳方法是什么？

Question

I have data being collected every 1 sec and stored in hsqlDB. I need to have aggregation data (per 15 sec, 1 min etc) on each metrics in the data collected. What is the best approach to calculate the aggregation values? When to store in the DB? Should I calculate the values online and each 15 sec store in DB? Or maybe query the DB for the last results and calculate the aggregation on them? Should I use small aggregation (15 sec) to calculate the large aggregation (1 min) ? Are there free java tools for it?

Answer 1

From previous experience, I would suggest using a real time database, probably non-relational with a built-in ability to deal with time series. That way, you should be able to avoid storing calculated aggregated data. Using a relational database, you will quickly end up with millions of rows that will be difficult to manage and slow to access. Your other option is to denormalize your data and store every 1 hour of data in a single row, in a BLOB column (in binary format).

Answer 2

You can use HSQLDB is MVCC mode for concurrent reads and writes.

Provided the table for the raw data has an indexed timestamp column, aggregate calculation on a range is very fast using a SELECT statement. Because SELECT statements with aggregate calculations happen concurrently, you can use separate threads to perform the operation every 1 second and every 15 seconds.