如何在Scala Spark中将RDD [Double]转换为矢量

Question

I have an IndexedRowMatrix of doubles. I want to compute the sum of each row of the matrix and save the results to a Vector. After that I want to broadcast this vector. I am creating an RDD of Doubles, which contains the sums, but I cannot turn it into a vector. So, the question basically is how to create the Vector I want from the IndexedRowMatrix.

Answer 1

Collect to the driver and construct a vector:

import org.apache.spark.mllib.linalg.{Vector, Vectors}

val sc: SparkContext = ???
val rdd: RDD[Double] = ???
val vec: Vector = Vectors.dense(rdd.collect)
val broadcastVec = sc.broadcast(vec)

References:

https://spark.apache.org/docs/2.1.0/mllib-data-types.html#local-vector https://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables