Filter rows based on a series pandas

Question

I've been facing a problem to filter out values in a column. I have a dataframe (data) which looks like the one below.

 Index                                                            Value
2019-11-22 00:00:00                                                0.0  
2019-11-22 00:05:00                                                1.0  
2019-11-22 00:10:00                                                2.0  
2019-11-22 00:15:00                                                3.0  
2019-11-22 00:20:00                                                4.0  
2019-11-22 00:25:00                                                5.0  
2019-11-22 00:30:00                                                6.0  
2019-11-22 00:35:00                                                7.0  
2019-11-22 00:40:00                                                8.0  
2019-11-22 00:45:00                                                0.0  
2019-11-22 00:50:00                                                0.0  
2019-11-22 00:55:00                                                1.0  
2019-11-22 01:00:00                                                2.0  
2019-11-22 01:05:00                                                3.0  
2019-11-22 01:10:00                                                4.0  
2019-11-22 01:15:00                                                5.0  

I want to keep the series of values which go above 5 and want to assign all others as zero. For example, if the values are from 1-5, all the previous values before 5 should be set to zero and if there are eight rows with values from 1-8, the code should keep them as it is.The final output should be the following.

 Index                                                            Value
2019-11-22 00:00:00                                                0.0  
2019-11-22 00:05:00                                                1.0  
2019-11-22 00:10:00                                                2.0  
2019-11-22 00:15:00                                                3.0  
2019-11-22 00:20:00                                                4.0  
2019-11-22 00:25:00                                                5.0  
2019-11-22 00:30:00                                                6.0  
2019-11-22 00:35:00                                                7.0  
2019-11-22 00:40:00                                                8.0  
2019-11-22 00:45:00                                                0.0  
2019-11-22 00:50:00                                                0.0  
2019-11-22 00:55:00                                                0.0  
2019-11-22 01:00:00                                                0.0  
2019-11-22 01:05:00                                                0.0  
2019-11-22 01:10:00                                                0.0  
2019-11-22 01:15:00                                                0.0  

When I try

    data[data<5]=0

It just returns the values higher than 5. Any help will be great on this.

Answer 1

Let's try this:

df = pd.read_clipboard(index_col=0, sep='\s\s+')

df.index = pd.to_datetime(df.index)

grp = df['Value'].diff().lt(0).cumsum()

df_out = df.where(df.groupby(grp)['Value'].transform('max').gt(5), 0)
print(df_out)

Output:

                     Value
Index                     
2019-11-22 00:00:00    0.0
2019-11-22 00:05:00    1.0
2019-11-22 00:10:00    2.0
2019-11-22 00:15:00    3.0
2019-11-22 00:20:00    4.0
2019-11-22 00:25:00    5.0
2019-11-22 00:30:00    6.0
2019-11-22 00:35:00    7.0
2019-11-22 00:40:00    8.0
2019-11-22 00:45:00    0.0
2019-11-22 00:50:00    0.0
2019-11-22 00:55:00    0.0
2019-11-22 01:00:00    0.0
2019-11-22 01:05:00    0.0
2019-11-22 01:10:00    0.0
2019-11-22 01:15:00    0.0
​

Answer 2

Try this:

filter = data["Value"].where(data["Value"] > 5, 0)
indices_with_6 = filter[filter == 6].index
for idx in indices_with_6:
    filter[idx - 5: idx] = [1., 2., 3., 4., 5.]
print(filter)

0     0
1     1
2     2
3     3
4     4
5     5
6     6
7     7
8     8
9     0
10    0
11    0
12    0
13    0
14    0
15    0
Name: Value, dtype: int64