Spark Word2Vec示例说明以及如何获得字符串之间的相似性

Question

I followed the example in the Spark documentation page to use word2vec, link . It worked but I didn't quite understand what it is trying to compute.

Are the output vectors the output strings representation?

If yes, I tried to compute the cosine similarity between them but I got negative values because the vectors are not positive.

Can Spark word2vec create positive only vectors?

How to compute similarity between a list of strings using Spark word2vec?

Answer 1

The output vector(by using transform on dataset) is a representation of the document(possibly sentence or sentences) which is supplied to the model .So; in essence this output is a combination of all the vector representation of each of the words in the given document(most likely a simple vector sum).

You can use findSynonyms to get "num" number of words closest in similarity to the given word. findSynonyms is based on cosine similarity only. Currently I am using it to generate feature Vectors which I am using as input to another model.

In order to compute similarity between two strings as some kind of a no. you would need to implement some variation of findSynonyms method.The current implementation generates a cosVec corresponding to input string and then tries to find the word Vecs which are closest to this vec .

I am not sure about the part whether it can create only positive vectors and whether it is at all required/(makes sense) to generate only positive vectors.