Reshaping only some columns into a single column

Question

import pandas as pd
from datetime import datetime

df = pd.DataFrame({'origin': ['japan', 'japan','japan','japan'],
                       'pastime': ['baseball', 'sumo', 'keirin', 'football'],
                       datetime(2000,1,1) : [4,5,4,5],
                   datetime(2005,1,1) : [4, 3, 2, 1],
                   datetime(2010,1,1) : [4, 2, 2, 1]
                  })

My dataframe has lots of date-labelled columns:

Index(['origin','pastime', 2000-01-01 00:00:00,
       2005-01-01 00:00:00, 2010-01-01 00:00:00],
      dtype='object')

I would like to reshape the dataframe to have columns: origin, pastime, date, value

Where the first entry row would be:

origin = japan
pastime = baseball
date = 2001-01-01
value = 4

I've seen examples using stack to push columns into the rows as an index, but in my case, it pushes the 'origin' and 'pastime' columns down too.

How would I do this transformation?

Answer 1

I think you're looking for melt :

df.melt(['origin', 'pastime'], var_name='date')

   origin   pastime       date  value
0   japan  baseball 2005-01-01      4
1   japan      sumo 2005-01-01      3
2   japan    keirin 2005-01-01      2
3   japan  football 2005-01-01      1
4   japan  baseball 2010-01-01      4
5   japan      sumo 2010-01-01      2
6   japan    keirin 2010-01-01      2
7   japan  football 2010-01-01      1
8   japan  baseball 2000-01-01      4
9   japan      sumo 2000-01-01      5
10  japan    keirin 2000-01-01      4
11  japan  football 2000-01-01      5

Answer 2

set_index and stack

df.set_index(['origin','pastime']).stack().reset_index()
Out[150]: 
   origin   pastime    level_2  0
0   japan  baseball 2010-01-01  4
1   japan  baseball 2000-01-01  4
2   japan  baseball 2005-01-01  4
3   japan      sumo 2010-01-01  2
4   japan      sumo 2000-01-01  5
5   japan      sumo 2005-01-01  3
6   japan    keirin 2010-01-01  2
7   japan    keirin 2000-01-01  4
8   japan    keirin 2005-01-01  2
9   japan  football 2010-01-01  1
10  japan  football 2000-01-01  5
11  japan  football 2005-01-01  1

PS. you can use rename for change the column name