带窗口函数的 SQL 中每个分区的 last_value 总和

Question

I have a table that stores total disk used at any point in time for each entity. I want to find the peak disk used in a time period. For example, the data looks something like

Note: The timestamp is actual timestamp with seconds precision, I set it to 10am etc for brevity

timestamp | entity_id | disk_used
---------------------------------
    9am   |         1 |  10
   10am   |         2 |  20
   11am   |         2 |  15
   12am   |         1 |  12
     

In this example, the max disk used at is 30 (10 from entity 1 and 20 from entity 2).

I have tried a number of approaches.

1. Sum of (max of each entity) does't work because it would give the result 20 + 12 = 32. But before the entity 1 increased its size, the entity 2 reduced the size, so the peak disk usage was 30.
2. I tried to use window function to find the sum of last_value of each entity

select timestamp, entity_id,
    disk_used, 
    sum(last_value(disk_used) over(
        partition by entity_id order by timestamp)
    ) sum_of_last

attempting to generate, so I can then max of it,

timestamp | entity_id | disk_used | sum_of_last
-----------------------------------------------
    9am   |         1 |  10       |   10
   10am   |         2 |  20       |   30
   11am   |         2 |  15       |   25       // (10 + 15)
   12am   |         1 |  12       |   27       // (12 + 15)
     

however, that query doesn't work because we cannot aggregate over a window function in ISO Standard SQL 2003. I am using Amazon timestream db. The query engine is compatible with ISO Standard SQL 2003.

-- Rephrasing the same question, at each timestamp we have the data point, for the total disk used at that instant. To find the total total disk used at that instant, sum the last value of each entity.

Is there an effective way to compute this?

Answer 1

If you have only two entities, you can do:

select t.*,
       (last_value(case when entity_id = 1 then disk_used end ignore nulls) over (order by time) +
        last_value(case when entity_id = 2 then disk_used end ignore nulls) over (order by time)
       ) as total        
from t;

One way to generalize this for all entities is to generate a row for each entity at each time, impute the value and aggregate:

select ti.time, e.entity_id,
       last_value(disk_used ignore nulls) over (partition by e.entity_id order by t.time) as imputed_disk_used
from (select distinct time from t) ti cross join
     (select distinct entity_id from t) e left join
     t
     on ti.time = t.time and e.entity_id = t.entity_id;

Then you can aggregate:

select time, sum(imputed_disk_used)
from (select ti.time, e.entity_id,
             last_value(disk_used ignore nulls) over (partition by e.entity_id order by t.time) as imputed_disk_used
      from (select distinct time from t) ti cross join
           (select distinct entity_id from t) e left join
           t
           on ti.time = t.time and e.entity_id = t.entity_id
     ) te
group by time;

However, this gives that value per time rather than per time and entity_id .

Answer 2

I want to find the peak disk used in a time period

You can use two levels of aggregation:

select max(sum_disk_used)
from (
    select time, sum(disk_used) as sum_disk_used
    from mytable
    group by time
) t

The subquery computest the total disk_used at each point in time, then the outer query gets the peak value only.

If your database supports some kind of limit clause, this can be simplified:

select time, sum(disk_used) as sum_disk_used
from mytable
group by time
order by sum_disk_used limit 1

To filter on a given period, you would typically add a where clause to the subquery.