Pyspark数据框到KMEANS的阵列RDD

Question

I am trying to run Kmeans clustering algo in Spark 2.2. I am not able to find the correct input format. It gives TypeError: Cannot convert type <class 'pyspark.sql.types.Row'> into Vector error. I checked further that my inputrdd is an Row Rdd. CAn we convert it to an array RDD? This MLlib Doc says shows that we can pass a paralleized array rdd data into the KMeans model. Error occurs at KMeans.train step.

import pandas as pd
from pyspark.mllib.clustering import KMeans, KMeansModel
df = pd.DataFrame({"c1" : [1,2,3,4,5,6], "c2": [2,6,1,2,4,6], "c3" : [21,32,12,65,43,52]})
sdf = sqlContext.createDataFrame(df)
inputrdd = sdf.rdd
model = KMeans.train( inputrdd, 2, maxIterations=10, initializationMode="random",
               seed=50, initializationSteps=5, epsilon=1e-4)

inputrdd when .collect is called.

[Row(c1=1, c2=2, c3=21),
 Row(c1=2, c2=6, c3=32),
 Row(c1=3, c2=1, c3=12),
 Row(c1=4, c2=2, c3=65),
 Row(c1=5, c2=4, c3=43),
 Row(c1=6, c2=6, c3=52)]

Answer 1

Following changes helped. I changed my Row rdd to Vector directly using Vectors.dense .

from pyspark.mllib.linalg import Vectors
inputrdd = sdf.rdd.map(lambda s : Vectors.dense(s))