简体   繁体   中英

Dropping rows from dataframe based on the value of the previous row

I have a dataframe like the following:

2018-10-16 15:15:00 1810.388020
2018-10-16 15:20:00 1813.299467
2018-10-16 15:25:00 1812.550495
2018-10-16 15:45:00 18140.981919
2018-10-16 15:35:00 1814.473347
2018-10-16 15:40:00 1816.349779
2018-10-16 15:45:00 740.981919
2018-10-16 15:50:00 1819.066781
2018-10-16 15:55:00 1820.333191

I need a dataframe like the following:

2018-10-16 15:15:00 1810.388020
2018-10-16 15:20:00 1813.299467
2018-10-16 15:25:00 1812.550495
2018-10-16 15:35:00 1814.473347
2018-10-16 15:40:00 1816.349779
2018-10-16 15:50:00 1819.066781
2018-10-16 15:55:00 1820.333191

That is:For every row having more value than 1.5 the previous row value or less than 0.5 the previous row value, drop it

Idea is compare previous and also next values, chaining by |for bitwise OR and last by bitwise AND with filtering by boolean indexing :

m11 = df['col'] > df['col'].shift(-1) * 0.5
m12 = df['col'] > df['col'].shift() * 0.5
m21 = df['col'] < df['col'].shift(-1) * 1.5
m22 = df['col'] < df['col'].shift() * 1.5

df = df[(m11 | m12) & (m21 | m22)]
print (df)
                             col
2018-10-16 15:15:00  1810.388020
2018-10-16 15:20:00  1813.299467
2018-10-16 15:25:00  1812.550495
2018-10-16 15:35:00  1814.473347
2018-10-16 15:40:00  1816.349779
2018-10-16 15:50:00  1819.066781
2018-10-16 15:55:00  1820.333191

Detail :

df['m11'] = df['col'] > df['col'].shift(-1) * 0.5
df['m12'] = df['col'] > df['col'].shift() * 0.5
df['m21'] = df['col'] < df['col'].shift(-1) * 1.5
df['m22'] = df['col'] < df['col'].shift() * 1.5

df['m1'] = (df['m11'] | df['m12'])
df['m2'] = (df['m21'] | df['m22'])

df['mask'] = df['m1'] & df['m2']

print (df)
                              col    m11    m12    m21    m22     m1     m2  \
2018-10-16 15:15:00   1810.388020   True  False   True  False   True   True   
2018-10-16 15:20:00   1813.299467   True   True   True   True   True   True   
2018-10-16 15:25:00   1812.550495  False   True   True   True   True   True   
2018-10-16 15:45:00  18140.981919   True   True  False  False   True  False   
2018-10-16 15:35:00   1814.473347   True  False   True   True   True   True   
2018-10-16 15:40:00   1816.349779   True   True  False   True   True   True   
2018-10-16 15:45:00    740.981919  False  False   True   True  False   True   
2018-10-16 15:50:00   1819.066781   True   True   True  False   True   True   
2018-10-16 15:55:00   1820.333191  False   True  False   True   True   True   

                      mask  
2018-10-16 15:15:00   True  
2018-10-16 15:20:00   True  
2018-10-16 15:25:00   True  
2018-10-16 15:45:00  False  
2018-10-16 15:35:00   True  
2018-10-16 15:40:00   True  
2018-10-16 15:45:00  False  
2018-10-16 15:50:00   True  
2018-10-16 15:55:00   True  

The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.

 
粤ICP备18138465号  © 2020-2024 STACKOOM.COM