将自定义函数应用于pandas数据框中的每个列组合

Question

I'm trying to calculate the cosine similarity between each combination of columns in my pandas dataframe. I've written a custom function to calculate cosine similarity, and now need to apply it to every combination pair of columns. Each column contains a 0 if a user has not interacted with it, and a 1 if the user has. Each row therefore contains the total viewing behaviour of the user.

Currently using a for loop, but its too slow for larger samples of data - eg my current sample is 3408 columns x 28000 rows.

My guess is a lambda function is the way to go, but I'm unsure how to apply it properly.

Initial dataframe:

sm_views = pd.read_sql(postgreSQL_select_Query, connection).groupby().size().unstack(fill_value=0)

Cos rating function:

def cos_rating_calculator(x, y):
    dot_product = np.dot(x, y)
    distance1 = np.sqrt(sum(x))
    distance2 = np.sqrt(sum(y))
    cos_rating = dot_product / (distance1 * distance2)
    return cos_rating

Code to calculate association combinations:

combinations = list(itertools.combinations(sm_views.columns, 2))

results = []

    for a, b in combinations:
        association_metric = cos_rating_calculator(sm_views[a], sm_views[b])
        results.append((a, b, association_metric))
        results.append((b, a, association_metric))

to_matrix = pd.DataFrame(results, columns=['a', 'b', 'association'])
association_matrix = to_matrix.pivot(index='a', columns='b', values='association')

For smaller datasets this works fine, however the current dataset is too large for this method to be feasible. My desired output is a column x column matrix with the degree of association between columns as values.

Answer 1

import scipy.spatial.distance
result = pd.DataFrame(list(itertools.combinations(sm_views.columns, 2)), columns=['a','b'])
result['association'] = scipy.spatial.distance.pdist(sm_views.T, 'cosine')

With this example sm_view:

   col1  col2  col3
0     0     0     0
1     3     4     2
2     1     1     5

we get

      a     b  association
0  col1  col2     0.002946
1  col1  col3     0.354058
2  col2  col3     0.414509