计算余弦相似度火花Java

Question

How to compute cosine similarity between 2 Spark Vector. I am using the new ml package.

Spark 2.1.1

EDIT:

Spark provide RowMatrix which can be used to compute similarity but it accepts mllib.vector not an ml.vector .

Is there a way to convert Vectors from the different packages? Is there an implementation that uses ml.vector ?

Answer 1

The easiest way to convert from an mllib vector to an ml vector is to use the Vectors.fromML method, see Vectors documentation . Example:

val mlVector = org.apache.spark.ml.linalg.Vectors.dense((Array(1.0,2.0,3.0)))
println(mlVector.getClass())

val mllibVector = org.apache.spark.mllib.linalg.Vectors.fromML(mlVector)
println(mllibVector.getClass())

Gives an output:

class org.apache.spark.ml.linalg.DenseVector
class org.apache.spark.mllib.linalg.DenseVector