将按列自定义 function 应用于 2 个数据帧，创建一个新的 dataframe

Question

I have 2 dataframes:

df_up = pd.DataFrame({"u1":[2,-3,5,0], 
                      "u2":[1,0,5,-2]},
                      index=["ta","tb","tc","td"])

df_tt = pd.DataFrame({"q1":[1,0,1,0], 
                      "q2":[1,0,1,1],
                      "q3":[0,1,0,0]},
                      index=["ta","tb","tc","td"])

I want to create a new dataframe that calculates the cosine similarity between all columns of df_up and all columns of df_tt. Both dataframes have the same number of rows.Ideally, the solution would work with a custom function, such as:

from scipy import spatial
def cosine_similarity(array_1, array_2):
    return 1 - spatial.distance.cosine(array_1,array_2)

The result would look like this:

    u1       u2
q1  0.8029   0.7745
q2  0.6556   0.4216
q3  -0.4866  0.0

Is there an "elegant" way of solving this or is iterating through the 2 dataframes the only way?

Answer 1

Solution from cdist

from scipy.spatial.distance import cdist
ary=(1-cdist(df_up.T.values, df_tt.T.values, metric='cosine')).T
df=pd.DataFrame(ary,columns=df_up.columns,index=df_tt.columns)
Out[258]: 
          u1        u2
q1  0.802955  0.774597
q2  0.655610  0.421637
q3 -0.486664  0.000000

Answer 2

A generic way is to use corr with a callable method, see below,

# There was a typo in the original method: array_1, array_2

def cosine_similarity(array1, array2):
    return 1 - spatial.distance.cosine(array1,array2)

output = (pd.concat([df_up, df_tt], axis=1)
            .corr(method=cosine_similarity)
            .drop(columns=df_tt.columns, index=df_up.columns))