将过去N天的日期行转换为列

Question

I want to build a time series prediction model using features such as week of the year, day of the week, season, etc.

Since the prediction will be highly affected by the most recent values, I want to use the values in the last 5 days, as features, however I am having trouble with data preparation for learning:

My current table looks like this:

    date        id  score
0   2014-01-01  A   75
1   2014-01-01  B   1
2   2014-01-01  C   2
4   2014-01-02  A   84
5   2014-01-02  B   1
6   2014-01-02  C   3
8   2014-01-03  A   1
9   2014-01-03  B   1
10  2014-01-03  C   1

So I want each row to look like this:

    date        id  score  date_1 date_2 date_3 date_4 date-5
10  2014-01-03  A   1      84     75     0      0      0 
 9  2014-01-03  B   1      1      1      0      0      0

Date_1 is the score of A, the day before its date on 'date' column, date_2 is two days before, and so on...

So that I can predict the next day, using the information of last 5 days and more features that are irrelevant to this question. It is OK to fill NaN values with 0

Answer 1

You can use groupby(id) and shift . You should have your df be sorted by date: df.sort_values('date') before using the following command:

for i in range(5):
    df['date_'+str(i+1)] = df.groupby('id')['score'].shift(i+1).fillna(0).astype(int)

Using the above command yields the following df:

Answer 2

Time shifting using Timedelta

The other answer is shifting by numeric index. Works in this instance, but it will break if there are gaps in the dates, or if the dates have not been sorted.

You can handle this by converting the DataFrame to a time series, then using the freq parameter of DataFrame.shift() with a pandas.Timedelta object.

Example data:

import pandas as pd
df = pd.DataFrame({'date': ['2014-01-01'] * 3 +
                           ['2014-01-02'] * 3 +
                           ['2014-01-03'] * 3,
                   'id': ['A', 'B', 'C'] * 3,
                   'score': [75, 1, 2, 84, 1, 3, 1, 1, 1]})
df.date = pd.to_datetime(df.date)
df.set_index('date', inplace=True)

The IDs mean we need a couple of loops to keep everything separate:

for i in range(5):
    for id in df.id.unique():
        col = 'date_{}'.format(i+1)
        freq = pd.Timedelta('{}d'.format(i+1))
        df.loc[df.id==id, col] = df.loc[df.id==id, 'score'].shift(freq=freq)
    df[col] = df[col].fillna(0).astype(int)

This produces same output as other approach on this example, but if you have a skip in the date it will be different.

Output:

           id  score  date_1  date_2  date_3  date_4  date_5
date                                                        
2014-01-01  A     75       0       0       0       0       0
2014-01-01  B      1       0       0       0       0       0
2014-01-01  C      2       0       0       0       0       0
2014-01-02  A     84      75       0       0       0       0
2014-01-02  B      1       1       0       0       0       0
2014-01-02  C      3       2       0       0       0       0
2014-01-03  A      1      84      75       0       0       0
2014-01-03  B      1       1       1       0       0       0
2014-01-03  C      1       3       2       0       0       0