使用 numpy 的变量之间的相关性

Question

I would need to calculate the correlation between the presence of uppercase, special punctuation, specific words, in texts labelled as fake/not fake.

For example:

Text      Label        Uppercase       Special Punctuation    Specific Word
text1       1                1                       0                   1
text2       0                0                       0                   0
text3       1                1                       1                   1
text4       1                1                       1                   1
text5       0                0                       0                   1

Uppercase, Special Punctuation and Specific Word can take only one of these two values: 1 or 0. I would like to determine the correlation between these features related to the label (fake=1/not fake=0). I thought to use Pearson correlation as follows

import numpy as np

# Create correlation matrix
corr_matrix = df.corr().abs()

May I ask you if this the right function to use or if there are different correlation functions in python that calculate the correlation between binary variables?

Answer 1

.corr() should work if you have numeric values.

If your variables are strings, just convert them into integers and use correlation. This should work:

df[['Uppercase','Special Punctuation', 'Specific Word']].astype(int).corr()

Answer 2

The function is correct, but I don't understand why you're using only absolute values. The sign of the correlation is informative for the direction of the association. I'm not familiar with your context so I'll just flag this without going any further.

Correlation can be calculated in subtly different ways, ie 'pearson', 'kendall', 'spearman'. The default method is 'pearson'. You can calculate using other methods by specifying the 'method' argument. Like this:

corr_matrix = df.corr(method = 'kendall')

more information can be found in the documentation here: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.corr.html