使用scikit-learn在项目描述之间的余弦相似度

Question

I am using python 2.7 and scikit-learn to find cosine similarity between item descriptions.

A have a df , for example:

items    description

1fgg     abcd ty
2hhj     abc r 
3jkl     r df

I did following procedures:

1) tokenizing and stemming each description

2) transform the corpus into vector space using tf-idf

3) calculated cosine distance between each description text as a measure of similarity. distance = 1 - cosinesimilarity(tfidf_matrix)

My goal is to have a similarity matrix of items like this and answer the question like: "What is the similarity between the items 1ffg and 2hhj :

        1fgg    2hhj    3jkl
1ffg    1.0     0.8     0.1
2hhj    0.8     1.0     0.0
3jkl    0.1     0.0     1.0 

How to get this result? Thank you for your time.

Answer 1

You can use numpy array to create the matrix and then add index and head to create a dataframe.

Assume you have a list of descriptions: descriptions = ['abc', 'bcd', 'etc' ...] and corresponding tf-idf matrix. (row number corresponds description number)

You want to create an empty numpy array of shape NxN, where N = len(words)

distance_matrix = np.zeros((N,N))

Then you need to fill it with actual distances:

for i in range(N):
    for j in range(N):
        distance_matrix[i,j] = cosine_distance(tf_idf[i,:], tf_idf[j,:])

You can create dataframe with

pandas.DataFrame(distance_matrix, index = items_list, columns = items_list)