database design - MySQL: How to store and split time series

Question

I have a table where I store historical data and add a record for items I'm tracking every 5 mins. This is an example using just 2 items:

+----+-------------+
| id |  timestamp  |
+----+-------------+
|  1 |  1533209426 |
|  2 |  1533209426 |
|  1 |  1533209726 |
|  2 |  1533209726 |
|  1 |  1533210026 |
|  2 |  1533210026 |
+----+-------------+

The problem is that I'm actually tracking 4k items and the table keeps getting bigger, also, I don't need 5 mins data if I want to get the last month. What I'm trying to understand is if there's a way to keep 5 mins records for the last 24h, 1h records for the last 7 days etc. Maybe every hour I could get the first 12 records from the 5 mins table and store the average in the 1h table? But what if some records are missing because there were errors? Is this the correct way to solve this problem or there are some better alternatives?

Answer 1

You are on the right track.

There are multiple issues to decide on how to handle -- missing entries, timestamps skewed by 1 second (or whatever), etc.

By providing a count (which should always be 12), you can discover some hiccups:

SELECT  FLOOR(timestamp / 3600) AS hr,  -- MEDIUMINT UNSIGNED
        COUNT(*),    -- TINYINT UNSIGNED
        AVG(metric)  -- FLOAT
    FROM tbl
    GROUP BY 1;

Yes, every hour, do the previous hour's worth of data. Add WHERE timestamp BETWEEN ... AND ... + 3599 to constrain the range in question. Then purge the same set of data.

The table would have PRIMARY KEY(hr) .

Unless you are talking about millions of rows in a table, I would not recommend any use of PARTITION .