How can I use melt() to reshape a pandas DataFrame to a list, creating an index from a crosstab column and creating a new variable in its place?

Question

I have a matrix of data 29523 rows x 503 cols of which 3 cols are indices (below is a subset for example).

IDX1|  IDX2  | IDX3 | 1983 Q4   |  X  | Y |  Z  |1984 Q1 |   X  | Y | Z 
---------------------------------------------------------------------------
A   |   A1   |  Q   |   10      |  A  | F | NaN | 110    |   A  | F | NaN
A   |   A2   |  Q   |   20      |  B  | C | 40  | 120    |   B  | C | 240
A   |   A3   |  Q   |   30      |  A  | F | NaN | 130    |   A  | F | NaN
A   |   A4   |  Q   |   40      |  B  | C | 80  | 140    |   B  | C | 280
A   |   A5   |  Q   |   50      |  A  | F | NaN | 150    |   A  | F | NaN
A   |   A6   |  Q   |   60      |  B  | F | 120 | 160    |   B  | F | 320

I read this into a DataFrame with:

>>> df = pd.read_csv(C:\filename.csv, low_memory=False, mangle_dupe_cols=False)

and then use pandas.melt() to pivot the data:

df1 = pd.melt(df, id_vars=['IDX1', 'IDX2', 'IDX3'], var_name='ValueType',
              value_name = 'Value')

I have also tried stack() but melt() proved better here.

IDX1    |   IDX2    |   IDX3    |   ValueType   |   Value
---------------------------------------------------------------
A       |   A1      |   Q       |   1983 Q4     |   10
A       |   A1      |   Q       |   X           |   A
A       |   A1      |   Q       |   Y           |   F
A       |   A1      |   Q       |   Z           |   NaN
A       |   A1      |   Q       |   1984 Q1     |   110
A       |   A1      |   Q       |   X           |   A
A       |   A1      |   Q       |   Y           |   F
A       |   A1      |   Q       |   Z           |   NaN
A       |   A2      |   Q       |   1983 Q4     |   20
A       |   A2      |   Q       |   X           |   B
A       |   A2      |   Q       |   Y           |   C
A       |   A2      |   Q       |   Z           |   40

The option mangle_dupe_cols on the read_csv if True will place a .int suffix against all ValueType s that are duplicated. This is not ideal, but without it there is no way of linking the values for the variables to the correct period.

What I would prefer to do is instead of having the Period (1984 Q1) as a ValueType , give the Period s corresponding Value a variable 'W' and have each period form part of the IDX as below:

IDX1    |   IDX2    |   IDX3 | IDX4    |    ValueType   |   Value
---------------------------------------------------------------
A       |   A1      |   Q    |  1983 Q4|    W           |   10
A       |   A1      |   Q    |  1983 Q4|    X           |   A
A       |   A1      |   Q    |  1983 Q4|    Y           |   F
A       |   A1      |   Q    |  1983 Q4|    Z           |   NaN
A       |   A1      |   Q    |  1984 Q1|    W           |   110
A       |   A1      |   Q    |  1984 Q1|    X           |   A
A       |   A1      |   Q    |  1984 Q1|    Y           |   F
A       |   A1      |   Q    |  1984 Q1|    Z           |   NaN
A       |   A2      |   Q    |  1983 Q4|    W           |   20
A       |   A2      |   Q    |  1983 Q4|    X           |   B
A       |   A2      |   Q    |  1983 Q4|    Y           |   C
A       |   A2      |   Q    |  1983 Q4|    Z           |   40

Is the above possible with pandas or numpy?

My final DataFrame is going to be 14,761,500 rows x 6 cols.

Answer 1

Given

In [189]: df
Out[189]: 
  IDX1 IDX2 IDX3  1983 Q4  X  Y    Z  1984 Q1 X.1 Y.1  Z.1
0    A   A1    Q       10  A  F  NaN      110   A   F  NaN
1    A   A2    Q       20  B  C   40      120   B   C  240
2    A   A3    Q       30  A  F  NaN      130   A   F  NaN
3    A   A4    Q       40  B  C   80      140   B   C  280
4    A   A5    Q       50  A  F  NaN      150   A   F  NaN
5    A   A6    Q       60  B  F  120      160   B   F  320

Let us first set ['IDX1', 'IDX2', 'IDX3'] as the index.

df = df.set_index(['IDX1', 'IDX2', 'IDX3'])

The other columns have a periodic quality to them; we want to handle every 4 columns as a group . This idea of "handling as a group" leads naturally to assigning a new index level to the column index; some value which is the same for every 4 columns. This would be ideal:

               1983 Q4            1984 Q1           
                     W  X  Y    Z       W  X  Y    Z
IDX1 IDX2 IDX3                                      
A    A1   Q         10  A  F  NaN     110  A  F  NaN
     A2   Q         20  B  C  240     120  B  C  240
     A3   Q         30  A  F  NaN     130  A  F  NaN
     A4   Q         40  B  C  280     140  B  C  280
     A5   Q         50  A  F  NaN     150  A  F  NaN
     A6   Q         60  B  F  320     160  B  F  320

We can achieve this by building a MultiIndex and assigning it to df.columns :

columns = [col for col in df.columns if col[0] not in set(list('XYZ'))]
df.columns = pd.MultiIndex.from_product([columns, list('WXYZ')])

Now the desired long-format DataFrame can be obtained by calling df.stack to move the column levels into the row index:

df.columns.names = ['IDX4', 'ValueType']
series = df.stack(['IDX4', 'ValueType'], dropna=False)

---

Note also that when mangle_dupe_cols=False , the duplicate columns, X , Y , Z , get overwritten . So you lose data with mangle_dupe_cols=False . For example, when you use mangle_dupe_cols=False the last row's Z value gets assigns to every Z column regardless of the period.

So we must use mangle_dupe_cols=True , (or just leave it out since that is the default) and adjust the code accordingly. That, fortunately, is not hard to do since we are reassigning df.columns to a custom-build MultiIndex anyway.

---

Putting it all together:

import numpy as np
import pandas as pd
df = pd.read_table('data', sep=r'\s*[|]\s*')
df = df.set_index(['IDX1', 'IDX2', 'IDX3'])
columns = [col for col in df.columns if col[0] not in set(list('XYZ'))]
df.columns = pd.MultiIndex.from_product([columns, list('WXYZ')])
df.columns.names = ['IDX4', 'ValueType']
series = df.stack(['IDX4', 'ValueType'], dropna=False)
print(series.head())

yields

IDX1  IDX2  IDX3  IDX4     ValueType
A     A1    Q     1983 Q4  W             10
                           X              A
                           Y              F
                           Z            NaN
                  1984 Q1  W            110
dtype: object

Note that since we've removed all the column levels, the result is a Series. If you want a DataFrame with 6 columns, then we should follow it up with:

series.name = 'Value'
df = series.reset_index()
print(df.head())

which yields

  IDX1 IDX2 IDX3     IDX4 ValueType Value
0    A   A1    Q  1983 Q4         W    10
1    A   A1    Q  1983 Q4         X     A
2    A   A1    Q  1983 Q4         Y     F
3    A   A1    Q  1983 Q4         Z   NaN
4    A   A1    Q  1984 Q1         W   110
...