Python Pandas self merge on previous data

Question

I have a DataFrame that contains many years worth of data. I want to make a couple columns containing the previous years' data from the same DataFrame. Here's an example:

df = pd.DataFrame({'id': [1,1,1,2,2,2,3,4,5,3,3,3,4],
                   'yr': [87,88,89,54,55,53,87,87,89,90,91,92,86],
                   'data': '1-87 1-88 1-89 2-54 2-55 2-53 3-87 4-87 5-89 3-90 3-91 3-92 4-86'.split()})
    data  id  yr
0   1-87   1  87
1   1-88   1  88
2   1-89   1  89
3   2-54   2  54
4   2-55   2  55
5   2-53   2  53
6   3-87   3  87
7   4-87   4  87
8   5-89   5  89
9   3-90   3  90
10  3-91   3  91
11  3-92   3  92
12  4-86   4  86

I'd like to add on another column that shows the previous years' data for that id number. like this:

    data  id  yr  last_year_data
0   1-87   1  87  NaN 
1   1-88   1  88  1-87
2   1-89   1  89  1-88
3   2-54   2  54  2-53
4   2-55   2  55  2-54
5   2-53   2  53  NaN
6   3-87   3  87  NaN
7   4-87   4  87  4-86
8   5-89   5  89  NaN
9   3-90   3  90  NaN
10  3-91   3  91  3-90
11  3-92   3  92  3-91
12  4-86   4  86  NaN

I tried to do this with a merge but I got Nan's all the way down in the 2nd half of the merge. Here's my code for that:

df['last_year'] = df['yr'].apply(lambda x: x-1 if x > 0 else None)
df_test = df.merge(df, how='left',indicator=False,left_on=['id','yr'],right_on=['id','last_year'])

I know there's a better way to do this, but I'm not sure what it is. can you help?

Answer 1

You can using shift

df['New']=df.sort_values(['id','yr']).groupby('id').data.shift()
df
Out[793]: 
    data  id  yr   New
0   1-87   1  87   NaN
1   1-88   1  88  1-87
2   1-89   1  89  1-88
3   2-54   2  54  2-53
4   2-55   2  55  2-54
5   2-53   2  53   NaN
6   3-87   3  87   NaN
7   4-87   4  87  4-86
8   5-89   5  89   NaN
9   3-90   3  90  3-87
10  3-91   3  91  3-90
11  3-92   3  92  3-91
12  4-86   4  86   NaN