将三个分离的rdd [org.apache.spark.mllib.linalg.Vector]火花化为单个rdd [Vector]

Question

i have three separated rdd[mllib....vectors] and i need to combine them as a one rdd[mllib vector].

val vvv = my_ds.map(x=>(scaler.transform(Vectors.dense(x(0))),Vectors.dense((x(1)/bv_max_2).toArray),Vectors.dense((x(2)/bv_max_1).toArray)))

more info: scaler => StandardScaler bv_max_... is nothing but the DenseVector from breeze lib in case for normalizing (x/max(x))

now i need to make them all as one i get ([1.],[2.],[3.]) and [[1.],[2.],[3.]] but i need [1.,2.,3.] as one vector

Answer 1

finally i found ... i dont know if this is the best.

i had 3d data set and i needed to perform x/max(x) normalization on two dimensions and apply standardScaler to another dimension. my problem was that in the end i had 3 separated Vectors like: eg [ [1.0],[4,0],[5.0] ] [ [2.0], [5.0], [6.0]]................but i needed [1.0,4.0,5.0] which can be passed to KMeans. i changed the above code as :

val vvv = dsx.map(x=>scaler.transform(Vectors.dense(x.days_d)).toArray ++ (x.freq_d/bv_max_freq).toArray ++ (x.food_d/bv_max_food).toArray).map(x=>Vectors.dense(x(0),x(1),x(2)))