根据功能结果，pandas groupby timeseries数据

Question

I am analyzing power systems time series data, and I am trying to find the contiguous data points that go beyond a certain threshold value.

I am currently using excel formula row by row manually to do this, but I as I am trying to search more efficient methods I realized that this could be done in python pandas groupby function.

However, as far as the examples that I have read, the groupby function only groups the rows if they have the same label. What I would like to do is to pass a certain function to groupby that could check if the value => 3, and then group those values, indexed by its starting and end time of breaching the threshold value => 3.

Input:

+-------+---------+------+
| Index |  Time   | Value|
+-------+---------+------+
|     0 | 00:00:01|   3  |
|     1 | 00:00:02|   4  |
|     2 | 00:00:03|   5  |
|     3 | 00:00:04|   2  |
|     4 | 00:00:05|   6  |
|     5 | 00:00:06|   7  |
|     6 | 00:00:07|   1  |
|     7 | 00:00:08|   9  |
+-------+---------+------+

Output:

+-------+-----------+----------+--------+
| Index | TimeStart | TimeEnd  | Value  |
+-------+-----------+----------+--------+
|     0 | 00:00:01  | 00:00:03 |  3,4,5 |
|     1 | 00:00:05  | 00:00:06 |  6,7   |
|     2 | 00:00:08  | 00:00:08 |  9     |
+-------+-----------+----------+--------+

Answer 1

• Create a mask where less than 3
• Cumulative sum to create groups where greater than or equal to 3
• filter the df by the mask, then groupby
• Use agg to pass several functions at once
• Rename columns

---

mask = df.Value.lt(3)
grp = mask.cumsum()

d1 = df[~mask].groupby(grp[~mask]).agg(dict(
    Time=['first', 'last'],
    Value=lambda x: ','.join(map(str, x))
))

d1.columns = ['TimeStart', 'TimeEnd', 'Value']

d1

      TimeStart   TimeEnd  Value
Value                           
0      00:00:01  00:00:03  3,4,5
1      00:00:05  00:00:06    6,7
2      00:00:08  00:00:08      9