Function works on each row of data frame, but not using df.apply

Question

I have this pandas dataframe containing two samples X and Y for each row:

import pandas as pd
import numpy as np
df = pd.DataFrame({'X': [np.random.normal(0, 1, 10),
                         np.random.normal(0, 1, 10),
                         np.random.normal(0, 1, 10)],
                   'Y': [np.random.normal(0, 1, 10),
                         np.random.normal(0, 1, 10),
                         np.random.normal(0, 1, 10)]})

I want to use a function ttest_ind() (a statistical test taking two samples as input) on each row, and take the first element of the response (the function returns two elements):

• If I do it for a given row, eg 1st row, it works:

   from scipy import stats stats.ttest_ind(df['X'][0], df['Y'][0], equal_var = False)[0] # Returns a float 

• However, if I use apply to do it on each row, I get an error:

   df.apply(lambda x: stats.ttest_ind(x['X'], x['Y'], equal_var = False)[0]) # Throws the following error: Traceback (most recent call last): File "pandas\\_libs\\index.pyx", line 154, in pandas._libs.index.IndexEngine.get_loc File "pandas\\_libs\\hashtable_class_helper.pxi", line 759, in pandas._libs.hashtable.Int64HashTable.get_item TypeError: an integer is required During handling of the above exception, another exception occurred: ... KeyError: ('X', 'occurred at index X') 

What am I doing wrong?

Answer 1

You just need to specify the axis on which you want to apply your function. Take a look at the relevant docs for apply() . In short, axis = 1 says "apply the function to each row of my dataframe". The default is axis = 0 , which tries to apply the function to each column instead.

df.apply(lambda x: stats.ttest_ind(x['X'], x['Y'], equal_var = False)[0], axis=1)

0    0.985997
1   -0.197396
2    0.034277