计算具有一定数量的 NaN 可接受的连续值

Question

There are several great answers for counting consecutive values that meet conditions, but I can't seem to find one that also permits a certain number of NaN.

For example, take the following dataframe:

Date           Val1
1900-01-01     NaN
1900-01-02     10
1900-01-03     11
1900-01-04     13
1900-01-05     NaN
1900-01-06     NaN
1900-01-07     17
1900-01-08     2
1900-01-09     NaN
1900-01-10     NaN
1900-01-11     2
1900-01-12     5
1900-01-13     6

Ideally, I want to count runs of a certain value with a certain number of NaNs acceptable. I can get the counts and run length for values, but how could I allow a certain number of NaN to be counted in the run?

In the above dataframe, if we permitted two NaNs and wanted values 10 or above, the run would start at 1900-01-01 and end at 1900-01-07, producing:

Date           Run length
1900-01-01     7

Note that run length is 7 as the first NaN is counted in the run.

I've tried creating two different columns counting both the length of the runs with proper values and the length of the runs with NaNs, but I'm unsure how to proceed. I know I can do it with pandas and I must be close, but just totally lost near the finish line!

Answer 1

Find where 'Val1' is notnull. Use that to locate the consecutive groups of NaN s but first mask the original DataFrame so we only count NaN rows.

m = df['Val1'].notnull()
s1 = df.where(~m).groupby(m.cumsum())['Date'].transform('count').le(2)

Together these two masks can be used to indicate True for 2 or fewer consecutive NaN s

(s1 & ~m)

0      True
1     False
2     False
3     False
4      True
5      True
6     False
7     False
8      True
9      True
10    False
11    False
12    False
dtype: bool

Combine that with your condition of >=10

gps = (s1 & ~m) | df['Val1'].ge(10)

Use this Series to group. Use where + dropna to get rid of all of the groups formed by things that do not meet the condition.

res = (df.where(gps).dropna(subset=['Date'])
         .groupby((~gps).cumsum())
         .agg(['first', 'count']))

#         Date        Val1      
#        first count first count
#0  1900-01-01     7  10.0     4
#1  1900-01-09     2   NaN     0

Finally, let's remove those groups that are based only on consecutive NaNs

res = res.loc[res[('Val1', 'count')].ne(0), 'Date']

#        first  count
#0  1900-01-01      7