如何获取 pandas DataFrame 中两列的每一行的 p 值？

Question

I would like to ask for any suggestion how to calculate p-value for each row in my pandas DataFrame. My dataframe looks like this - there are columns with means of Data1 and Data2, and then also columns with standard error of the means. Each row represent one atom. Thus I need calculate p-value for each row (= it means, eg, compare mean of atom 1 from Data1 with mean of atom 1 from Data2).

    SEM-DATA1   MEAN-DATA1  SEM-DATA2   MEAN-DATA2  
0   0.001216    0.145842    0.000959    0.143103    
1   0.002687    0.255069    0.001368    0.250505    
2   0.005267    0.321345    0.003722    0.305767    
3   0.027265    0.906731    0.033637    0.731638    
4   0.029974    0.773725    0.150025    0.960804        

I found here on Stack that many people recommend using scipy. But I dont know how to apply it in the way I need it. Is it possible? Thank You.

Answer 1

You are comparing two samples df['MEAN...1'] and df['MEAN...2'] , so, you should do this:

from scipy import stats
stats.ttest_ind(df['MEAN-DATA1'],df['MEAN-DATA2'])

which return:

Ttest_indResult(statistic=0.01001479441863673, pvalue=0.9922547232600507)

or if you only want to p-value

a = stats.ttest_ind(df['MEAN-DATA1'],df['MEAN-DATA2'])
a[1]

which gives

0.9922547232600507

A clarification is in order here. A t-test (or the aquisition of a "p-value" is aimed at finding out is two distributions are coming from the same population (or sample). Testing for two single values will give NaN .