[英]remove all rows in pandas dataframe with N or more consecutive NaNs
corollary to this question: replace values in pandas column when N number of NaNs exist in another column 这个问题的推论: 当另一列中存在N个NaN时,替换pandas列中的值
a b c d e
2018-05-25 0.000381 0.264318 land 2018-05-25
2018-05-26 0.000000 0.264447 land 2018-05-26
2018-05-27 0.000000 0.264791 NaN NaT
2018-05-28 0.000000 0.265253 NaN NaT
2018-05-29 0.000000 0.265720 NaN NaT
2018-05-30 0.000000 0.266066 land 2018-05-30
2018-05-31 0.000000 0.266150 NaN NaT
2018-06-01 0.000000 0.265816 NaN NaT
2018-06-02 0.000000 0.264892 land 2018-06-02
2018-06-03 0.000000 0.263191 NaN NaT
2018-06-04 0.000000 0.260508 land 2018-06-04
2018-06-05 0.000000 0.256619 NaN NaT
2018-06-06 0.000000 0.251286 NaN NaT
2018-06-07 0.000000 0.244250 NaN NaT
2018-06-08 0.000000 0.235231 NaN NaT
2018-06-09 0.000000 0.223932 land 2018-06-09
I want to remove all rows where there is a NaN in the 4th column ( d
) 3 or more times. 我想删除第4列(
d
)中有NaN的所有行3次或更多次。 The output should be: 输出应该是:
a b c d e
2018-05-25 0.000381 0.264318 land 2018-05-25
2018-05-26 0.000000 0.264447 land 2018-05-26
2018-05-30 0.000000 0.266066 land 2018-05-30
2018-05-31 0.000000 0.266150 NaN NaT
2018-06-01 0.000000 0.265816 NaN NaT
2018-06-02 0.000000 0.264892 land 2018-06-02
2018-06-03 0.000000 0.263191 NaN NaT
2018-06-04 0.000000 0.260508 land 2018-06-04
2018-06-09 0.000000 0.223932 land 2018-06-09
From that question, I tried this: 从那个问题,我试过这个:
threshold = 3
mask = df.d.notna()
df.loc[(~mask).groupby(mask.cumsum()).transform('cumsum') < threshold, 'c'] = np.nan
df = df[np.isfinite(df['c'])]
but it does not work 但它不起作用
Create helper Series
a by consecutive values and transform
size
, last filter by boolean indexing
: 通过连续值和
transform
size
创建辅助Series
a,通过boolean indexing
创建最后一个过滤器:
mask = df.d.notna()
a = mask.ne(mask.shift()).cumsum()
df = df[(a.groupby(a).transform('size') < 3) | mask]
print (df)
a b c d e
0 2018-05-25 0.000381 0.264318 land 2018-05-25
1 2018-05-26 0.000000 0.264447 land 2018-05-26
5 2018-05-30 0.000000 0.266066 land 2018-05-30
6 2018-05-31 0.000000 0.266150 NaN NaT
7 2018-06-01 0.000000 0.265816 NaN NaT
8 2018-06-02 0.000000 0.264892 land 2018-06-02
9 2018-06-03 0.000000 0.263191 NaN NaT
10 2018-06-04 0.000000 0.260508 land 2018-06-04
15 2018-06-09 0.000000 0.223932 land 2018-06-09
Detail : 细节 :
print (a)
0 1
1 1
2 2
3 2
4 2
5 3
6 4
7 4
8 5
9 6
10 7
11 8
12 8
13 8
14 8
15 9
Name: d, dtype: int32
print (a.groupby(a).transform('size'))
0 2
1 2
2 3
3 3
4 3
5 1
6 2
7 2
8 1
9 1
10 1
11 4
12 4
13 4
14 4
15 1
Name: d, dtype: int64
This should work: 这应该工作:
df = df.groupby(pd.notnull(df.d).cumsum()).apply(lambda x: x.dropna() if pd.isnull(x.d).sum() >= 3 else x).reset_index(drop=True)
Output: 输出:
a b c d e
0 2018-05-25 0.000381 0.264318 land 2018-05-25
1 2018-05-26 0.000000 0.264447 land 2018-05-26
2 2018-05-30 0.000000 0.266066 land 2018-05-30
3 2018-05-31 0.000000 0.266150 NaN NaT
4 2018-06-01 0.000000 0.265816 NaN NaT
5 2018-06-02 0.000000 0.264892 land 2018-06-02
6 2018-06-03 0.000000 0.263191 NaN NaT
7 2018-06-04 0.000000 0.260508 land 2018-06-04
8 2018-06-09 0.000000 0.223932 land 2018-06-09
one way to solve this, 解决这个问题的一种方法,
df['seq'] = df.groupby(df['d'].notnull().cumsum())['a'].transform(len)
df=df[(df['seq']<4)|df['d'].notnull()]
Output: 输出:
a b c d e seq
0 2018-05-25 0.000381 0.264318 land 2018-05-25 1
1 2018-05-26 0.000000 0.264447 land 2018-05-26 4
5 2018-05-30 0.000000 0.266066 land 2018-05-30 3
6 2018-05-31 0.000000 0.266150 NaN NaN 3
7 2018-06-01 0.000000 0.265816 NaN NaN 3
8 2018-06-02 0.000000 0.264892 land 2018-06-02 2
9 2018-06-03 0.000000 0.263191 NaN NaN 2
10 2018-06-04 0.000000 0.260508 land 2018-06-04 5
15 2018-06-09 0.000000 0.223932 land 2018-06-09 1
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.