如何在熊猫日期时间序列的开始/结束位置找到连续的NAN

Question

I have a time series mapping each day to the number of steps walked. I would like to fill missing values one way if they appear consecutively in the beginning of the date-range I am examining. (And otherwise if they are in the "middle" of the data or at the end). Is there a way similar to str.startswith for identifying these consecutive NANs?

Here's an example:

original dataset:

               ID  Steps
Day                     
2019-07-25  53668    0.0
2019-07-26  53668    0.0
2019-07-27  53668    0.0
2019-07-28  53668  100.0
2019-07-29  53668    0.0
2019-07-30  53668    0.0
2019-07-31  53668    0.0
2019-08-01  53668  100.0
2019-08-02  53668    0.0
2019-08-03  53668    0.0
2019-08-04  53668    0.0
2019-08-05  53668    0.0

idx = pd.date_range('2019-07-20','2019-08-03')
df.reindex(idx, fill_value = np.nan)

yields:

                 ID  Steps
2019-07-20      NaN    NaN
2019-07-21      NaN    NaN
2019-07-22      NaN    NaN
2019-07-23      NaN    NaN
2019-07-24      NaN    NaN
2019-07-25  53668.0    0.0
2019-07-26  53668.0    0.0
2019-07-27  53668.0    0.0
2019-07-28  53668.0  100.0
2019-07-29  53668.0    0.0
2019-07-30  53668.0    0.0
2019-07-31  53668.0    0.0
2019-08-01  53668.0  100.0
2019-08-02  53668.0    0.0
2019-08-03  53668.0    0.0

How do I know that the 28 NaNs here are at the beginning and not interspersed, or at the end?

Answer 1

No, there si no such function.

You need write it:

#change data for NaNs in middle
print (df)
               ID  Steps
Day                     
2019-07-25  53668    0.0
2019-07-26  53668    0.0
2019-07-27  53668    0.0
2019-07-28  53668  100.0
2019-07-31  53668    0.0
2019-08-01  53668  100.0
2019-08-02  53668    0.0
2019-08-03  53668    0.0
2019-08-04  53668    0.0
2019-08-05  53668    0.0

---

idx = pd.date_range('2019-07-20','2019-08-08')
df = df.reindex(idx, fill_value = np.nan)
print (df)
                 ID  Steps
2019-07-20      NaN    NaN
2019-07-21      NaN    NaN
2019-07-22      NaN    NaN
2019-07-23      NaN    NaN
2019-07-24      NaN    NaN
2019-07-25  53668.0    0.0
2019-07-26  53668.0    0.0
2019-07-27  53668.0    0.0
2019-07-28  53668.0  100.0
2019-07-29      NaN    NaN
2019-07-30      NaN    NaN
2019-07-31  53668.0    0.0
2019-08-01  53668.0  100.0
2019-08-02  53668.0    0.0
2019-08-03  53668.0    0.0
2019-08-04  53668.0    0.0
2019-08-05  53668.0    0.0
2019-08-06      NaN    NaN
2019-08-07      NaN    NaN
2019-08-08      NaN    NaN

---

m = df['ID'].isna()
first = df['ID'].ffill().isna()
last = df['ID'].bfill().isna()

print (df[first])

2019-07-20 NaN    NaN
2019-07-21 NaN    NaN
2019-07-22 NaN    NaN
2019-07-23 NaN    NaN
2019-07-24 NaN    NaN

print (df[last])
            ID  Steps
2019-08-06 NaN    NaN
2019-08-07 NaN    NaN
2019-08-08 NaN    NaN

print (df[~(first | last) & m])
            ID  Steps
2019-07-29 NaN    NaN
2019-07-30 NaN    NaN