熊猫：向前填充而不填充尾随的NaN

Question

I have a dataframe where each column is a time series of different length. As such, there are missing values both between values in the time series, and at the end of each time series save one column. I would like to fill the missing values between values, but not fill the "trailing" NaNs

Using df = df.fillna(method='ffill') gets me most of the way there, but fills in the trailing NaNs, which I don't want, because where the data ends is actually important to my analysis.

Edit:

I would like to turn this:

            ERICB SS Equity  DCI US Equity  FLEX US Equity
date

2008-02-14            8.026            NaN             NaN
2008-02-18              NaN            NaN           1.472
2008-02-19            8.074            NaN             NaN
2008-02-22              NaN            NaN           1.532
2008-02-25            8.062            NaN             NaN
2008-03-03            8.100            NaN             NaN
2008-03-06            8.100            NaN           1.955
2008-03-07            8.100            NaN             NaN
2010-12-30            5.431            NaN             NaN
2010-12-31            5.422            NaN             NaN
2011-01-03            5.422            NaN             NaN
2011-01-04            5.373            NaN             NaN

Into this:

            ERICB SS Equity  DCI US Equity  FLEX US Equity
date

2008-02-14            8.026            NaN             NaN
2008-02-18            8.026            NaN           1.472
2008-02-19            8.074            NaN           1.472
2008-02-22            8.074            NaN           1.532
2008-02-25            8.062            NaN           1.532
2008-03-03            8.100            NaN           1.532
2008-03-06            8.100            NaN           1.955
2008-03-07            8.100            NaN             NaN
2010-12-30            5.431            NaN             NaN
2010-12-31            5.422            NaN             NaN
2011-01-03            5.422            NaN             NaN
2011-01-04            5.373            NaN             NaN

So it's forward filled but only when there is some non-null value in the future to fill to, leaving the trailing nulls.

Answer 1

One way would be to bfill, which makes everything before the last non-NaN value non-NaN, and then use where to select the ffill() results:

In [45]: df.ffill().where(df.bfill().notnull())
Out[45]: 
          date  ERICB SS Equity  DCI US Equity  FLEX US Equity
0   2008-02-14            8.026            NaN             NaN
1   2008-02-18            8.026            NaN           1.472
2   2008-02-19            8.074            NaN           1.472
3   2008-02-22            8.074            NaN           1.532
4   2008-02-25            8.062            NaN           1.532
5   2008-03-03            8.100            NaN           1.532
6   2008-03-06            8.100            NaN           1.955
7   2008-03-07            8.100            NaN             NaN
8   2010-12-30            5.431            NaN             NaN
9   2010-12-31            5.422            NaN             NaN
10  2011-01-03            5.422            NaN             NaN
11  2011-01-04            5.373            NaN             NaN

Another would be to directly make a mask containing True for all values up to and including the last valid value:

df.ffill().where(df.notnull().iloc[::-1].cummax().iloc[::-1])

where the .iloc[::-1] stuff is required because I can't find a better way to take a cumulative operation in the bottom-to-top direction.