Mean/Std calculation of data-set logging change of value only

Question

I wish to calculate the mean, and std of a time series data-set that only records when there is a change in value.

The time series is at 1 second intervals, which puts the risk of a bloated data-set if each missing value between changes is generated.

Example of data set type: [Epoch, Value]

[[152345235, 3], [152345335, 12], [152345835, 8]]

We are using python for cleaning and aggregating the data. A sparse matrix would not suffice, as it is based on ignoring 0 values, which are not present. On research there was nothing showing that could readily solve this problem.

My question is, has anyone come across a similar problem, and if so what techniques were used to solve.

Answer 1

I would order the list by epoch , then multiply the value of each by the difference between epochs . You don't need to store extra data that way, but you can pretty easily solve for the mean.