如何将高维数据框从长到宽重塑，以便后续降维+可视化？

Question

I have a data frame that resembles the following:

This looks like the following:

index	attribute	score
user_1	a	0.144228
user_1	b	0.980685
user_1	c	0.165716
user_2	a	0.795340
user_2	b	0.903498
user_3	d	0.193492
user_3	e	0.900509

Here's the reproducible code:

df = pd.DataFrame({'index':['user_1','user_1','user_1','user_2','user_2','user_3','user_3'],
                   'attribute':['a','b','c','a','b','d','e'],
              'score':[random.rand(),random.rand(),random.rand(),random.rand(),random.rand(),random.rand(),random.rand()]})


df.set_index('index',inplace=True)

I'd like to unstack/pivot this table so that the attribute values becomes column header, like so:

Now, this is fairly easy, except, I have 350K dimensions , and as you can see from the above example, not every user has scores for each dimension .

I've tried using the standard pandas pd.pivot_table() and .unstack() functions, but my kernel invariably dies when I attempt to do so. I subsequently attempted to do so using dask, saving the output to a csv via

dask.dataframe.reshape.pivot_table(df, index='index', columns='attribute', values='score').to_csv('df.csv')

but that crashed too, yielding the following error:

KilledWorker: ("('pivot_table_count-chunk-c31649485f27d5f8670393d66e2d14ac', 0, 3, 0)", <Worker 'tcp://127.0.0.1:56298', name: 0, memory: 0, processing: 5>)

I'm currently at a loss. How can I reshape high-dimensional dataset for subsequent dimension reduction, clustering, and viz?

Answer 1

df.pivot(index='index', columns='attribute').reset_index().droplevel(0, axis=1)


           attribute    a         b         c         d         e
0          user_1  0.144228  0.980685  0.165716       NaN       NaN
1          user_2  0.795340  0.903498       NaN       NaN       NaN
2          user_3       NaN       NaN       NaN  0.193492  0.900509