使用 Pandas 计算一个值超过限制的总时间

Question

Given a column of datetime, a column of value and a limit, I have to calculate the total amount of time that goes over the limit. Sample data looks like this: let's say limit is 30

Time                    Value
2018-01-03 12:54:23     23
2018-01-03 12:58:46     31
2018-01-03 13:02:12     32
2018-01-03 13:04:13     24
2018-01-03 13:07:01     28

My idea is first to use shift function to calculate the time differences between each timestamp. Then use a for loop to goes over the value. If the previous and current value are both over the limit, then we increase the total time by the time differences

temp["TimeDifference"] = (temp.Time -temp.Time.shift(1)).fillna(pd.Timedelta(seconds=0))

total_time = pd.Timedelta(seconds=0)

for i in range(1, temp.shape[0]):
    if (temp.loc[i - 1].Value > upper_limit) and (temp.loc[i].Value > upper_limit):
        total_time = total_time + temp.loc[i].TimeDifference

It works... but the run time is really long and I know that this algorithm isn't efficient. Can someone give my an advise? Thanks

Answer 1

You cite the shift function, but then you didn't use it. Instead, you hard-coded the -1 shift and wrote your own loop to walk through the rows.

Instead, do something like this, to use the built-in vectorization.

sum((temp.Time - temp.Time.shift(1)) if temp.Value > 30)

... but I need someone to check my syntax; this is top-of-the-head coding between urgent tasks.