Postgres copy data from one table to another periodically by applying crosstab

Question

Consider I have a web app wherein I show current data values coming from sensors of sensor data. Assume we have 2 sensors, sensor A and sensor B, with ids values 1,2 respectively Assume we have 2 tags each temperature and humidity.

I have configured a Nodejs app to pull data from sensors every 500 milliseconds and push data into postgres tables "data_live" like below.

sensor_id |tag          |value       
----------+-------------+------------
         1|temperature  |0.006817675 
         1|humidity     |0.002902401 
         2|temperature  |33          
         2|humidity     |28         

Note. Here, in table "data_live" we just keep current value for each machine/tag so every time we push update, we do update operation on database.

I want to record the history in timeseries manner using timescale extension in the below table named "data_ts".

time                         |machine_id|temperature|humidity
-----------------------------+----------+-----------+--------
2022-04-09 14:19:01.000 +0530|         1|       20.2|    55.3
2022-04-09 14:19:01.000 +0530|         2|       19.7|    50.1
2022-04-09 14:19:02.000 +0530|         1|       20.3|    55.4
2022-04-09 14:19:02.000 +0530|         2|       19.6|    50.0

I am thinking of using cron based scheduler to run a script periodically to perform below steps:

1. Fetching data from table: "data_live"
2. Applying crosstab (convert tabular structure to columnar structure) function
3. Insert columnar data into table "data_ts"

Limitation with this approach is I cannot execute cron scheduler every 500ms. Sensors can go upto 100+ and tags can go upto 50+, So we also have to think about the scale.

Can anyone suggest me the solution here? Let me know if you need any more information. Thanks in advance.

Answer 1

If you want the solution to be self-contained in PG, the cron option is valid. However, since you call the scale problem, I would suggest you start thinking about data retention as well. Eg for how long you want to keep the data for and have another job that periodically cleans the old entries.

If you want to expand the tech horizon, there are a multitude of alternative solution, some of which have been identified by the comment from @webdev_jj