流Kmeans Spark JAVA

Question

Hi Basically we wanted to use KAFKA+SPARK Streaming to catch Twitter Spam on our thesis. And I wanted to use streamingKmeans. But I have very newbie and serious question:

In this spark StreamingKmeans scala example ( https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples/mllib/StreamingKMeansExample.scala ) there is one line of code for prediction:

model.predictOnValues(testData.map(lp => (lp.label, lp.features))).print()

Why I need to pass the "LABEL" with features ? I mean, am I getting wrong the whole idea ? Isn't we want to predict the "label" ? How am I going to predict my tweets if they are spam or not ?

Answer 1

For the prediction only lp.features is used, whereas lp.label is considered as a key that is carried over. Quoting from the docs :

Use the model to make predictions on the values of a DStream and carry over its keys.

I guess in your example you would simply want to replace predictOnValues by predictOn