How to test correlation between two sets in python?

Question

I have two different dataframe and one of them is as below

df1=

      Datetime      BSL
0          7  127.504505
1          8  115.254132
2          9  108.994275
3         10  102.936860
4         11   99.830400
5         12  114.660522
6         13  138.215339
7         14  132.131075
8         15  121.478006
9         16  113.795645
10        17  114.038462

the other one is df2=

    Datetime       Number of Accident
0          7                  3455
1          8                 17388
2          9                 27767
3         10                 33622
4         11                 33474
5         12                 12670
6         13                 28137
7         14                 27141
8         15                 26515
9         16                 24849
10        17                 13013

the first one Blood Sugar Level of people based on time (7 means between 7 am and 8 am) the second one is number of accident between these times

when I try to this code

df1.corr(df2, "pearson")

I got as error:

ValueError: The truth value of a DataFrame is ambiguous. Use a.empty, a.bool(), a.item(), a.any() or a.all().

How can I solve it? Or, how can I test correlation between two different variables?

Answer 1

from scipy.stats import pearsonr
df_full = df1.merge(df2,how='left')
full_correlation = pearsonr(df_full['BSL'],df_full['Accidents'])
print('Correlation coefficient:',full_correlation[0])
print('P-value:',full_correlation[1])

Output:

(-0.2934597230564072, 0.3811116115819819)
Correlation coefficient: -0.2934597230564072
P-value: 0.3811116115819819

Edit:

You want an hourly correlation, but it is impossible mathematically because you have only 1 xy value for each hour. Therefore the output will be full of NaNs. This is the code, however the output is invalid:

df_corr = df_full.groupby('Datetime')['BSL','Accidents'].corr().drop(columns='BSL').drop('Accidents',level=1).rename(columns={'Accidents':'Correlation'})
print(df_corr)

Output:

              Correlation
Datetime                 
7        BSL          NaN
8        BSL          NaN
9        BSL          NaN
10       BSL          NaN
11       BSL          NaN
12       BSL          NaN
13       BSL          NaN
14       BSL          NaN
15       BSL          NaN
16       BSL          NaN
17       BSL          NaN

Answer 2

由于您的数据框有多个列，您需要指定要使用的列的名称：

df1['BSL'].corr(df2['Number of Accident'], "pearson")

Answer 3

The corr() method of a pandas dataframe calculates a correlation matrix for all columns in one dataframe. You have two dataframes, so that method won't work. You can solve this by doing:

df1['number'] = df2['Number of Accident']
df1.corr("pearson")