python时间序列滞后于shift（1），如何为创建的NaN填充

Question

I have aa very large dataset containing id and data points of time series (with some missing values). The following is just an example.

I will need to create a lag variable for both group which of course will create NaN for the first observation for each group. I would like to assign the next available value to the created NaN specifically but leave other missing value untouched for later manipulation.

id    time    value    lag_value 
A     2000    10       NaN      # I want this to be 10, the next available value 
A     2001    11       10 
A     2002    NaN      11 
A     2003    14       NaN 
A     2004    10       14 

Edit:

I think it would be cleaner to use first_valid_index to assign the next available value, see Pandas - find first non-null value in column

Answer 1

Here you go, this will fill the first value with the first non NaN entry from the original list.

import pandas as pd
import numpy as np
df = pd.DataFrame({'id': ['A', 'A', 'A', 'A', 'A'],
                  'time': [2000, 2001, 2002, 2003, 2004],
                  'value': [10, 11, np.NaN, 14, 10]})

df['lag_value'] = df.value.shift(1)
df.loc[0, 'lag_value'] = df.lag_value[df.lag_value.notnull()].values[0]

#  id  time  value  lag_value
#0  A  2000   10.0       10.0
#1  A  2001   11.0       10.0
#2  A  2002    NaN       11.0
#3  A  2003   14.0        NaN
#4  A  2004   10.0       14.0

Answer 2

Since you mention first_valid_index

s=df.value.shift()
s.iloc[s.first_valid_index()-1]=df.value.iloc[0]
s
Out[110]: 
0    10.0
1    10.0
2    11.0
3     NaN
4    14.0
Name: value, dtype: float64