按行创建相关矩阵：Pandas

Question

I want to create a correlation matrix by rows. Here's how my df looks like:

df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]),
                   columns=['a', 'b', 'c'],index = ["doc1", "doc2", "doc3"])

#Output
      a  b  c
doc1  1  2  3
doc2  4  5  6
doc3  7  8  9

I want to find the correlation between documents. I used

corrMatrix = df.corr()

but it gives me correlation between each cell (I think). The other approach that I have considered is to simply subset each of the document and then use

np.corrcoef(doc1,doc2)

and manually create a 2D numpy array. Any ideas where I can do this elegantly?

Answer 1

DataFrame.corr() finds the correlation between pairs of columns . If you want rows, transpose first. (I modified your data slightly so everything isn't perfectly correlated)

import pandas as pd
import numpy as np
df = pd.DataFrame(np.array([[1, 2, 8], [4, 5, 6], [5, 8, 9]]),
                  columns=['a', 'b', 'c'], index=["doc1", "doc2", "doc3"])

---

df.T.corr()

          doc1      doc2      doc3
doc1  1.000000  0.924473  0.782467
doc2  0.924473  1.000000  0.960769
doc3  0.782467  0.960769  1.000000

---

Or use np.corrcoef on the non-transposed DataFrame. This will be a lot faster than the above with a large DataFrame since you avoid the unnecessary transpose.

np.corrcoef(df)

array([[1.        , 0.92447345, 0.78246663],
       [0.92447345, 1.        , 0.96076892],
       [0.78246663, 0.96076892, 1.        ]])