Pandas - How to shift a column based on other columns values

Question

I've a big Pandas dataset (46 million rows), represented here by a little sample:

    df = pd.DataFrame([[0, 0, 0, 34],[0, 0, 1, 23],[0, 1, 0, 14],[0, 1, 1, 11],[1, 0, 0, 73],[1, 0, 1, 33],[1, 1, 0, 96],[1, 1, 1, 64],[2, 0, 0, 4],[2, 0, 1, 13],[2, 1, 0, 31],[2, 1, 1, 10]])

df.columns = ['month','player','team','skill']

For each month we have a product cartesian of players and teams

id month player team skill
0   0   0   0   34
1   0   0   1   23
2   0   1   0   14
3   0   1   1   11
4   1   0   0   73
5   1   0   1   33
6   1   1   0   96
7   1   1   1   64
8   2   0   0   4
9   2   0   1   13
10  2   1   0   31
11  2   1   1   10

I would like to shift the skill column backwords by month, in order to get something like this

0   0   0   0   73
1   0   0   1   33
2   0   1   0   96
3   0   1   1   64
4   1   0   0   4
5   1   0   1   13
6   1   1   0   31
7   1   1   1   10
8   2   0   0   Nan
9   2   0   1   Nan
10  2   1   0   Nan
11  2   1   1   Nan

How can I do this in Pandas efficiently? Thanks!

Answer 1

If I understand you correctly, you want to find the skill for the same player-team combination in the following month. You can do that with groupby and transform :

# Sort the rows by `player-team-month` combination so that the
# next row is the subsequent month for the same `player-team`
# or a new `player-team`
tmp = df.sort_values(['player', 'team', 'month'])

# The groupby here serves to divide the dataframe by `player-team`
# Each group is now ordered by `month` so `skill.shift(-1)` can
# give us the `skill` in the following month
skill = tmp.groupby(['player', 'team'])['skill'].transform(lambda s: s.shift(-1))

# Combine the shifted skill with the original attributes
result = pd.concat([tmp[['month', 'player', 'team']], skill], axis=1)