使用保存的 Spark 模型评估新数据

Question

I've been successful in building a transforming my data into a LibSVM file, and training a decision tree model on it in Spark's MLlib package. I used the Scala code in the 1.6.2 documentation , changing only the filenames:

import org.apache.spark.mllib.tree.DecisionTree
import org.apache.spark.mllib.tree.model.DecisionTreeModel
import org.apache.spark.mllib.util.MLUtils

// Load and parse the data file.
val data = MLUtils.loadLibSVMFile(sc, "data/mllib/sample_libsvm_data.txt")
// Split the data into training and test sets (30% held out for testing)
val splits = data.randomSplit(Array(0.7, 0.3))
val (trainingData, testData) = (splits(0), splits(1))

// Train a DecisionTree model.
//  Empty categoricalFeaturesInfo indicates all features are continuous.
val categoricalFeaturesInfo = Map[Int, Int]()
val impurity = "variance"
val maxDepth = 5
val maxBins = 32

val model = DecisionTree.trainRegressor(trainingData, categoricalFeaturesInfo, impurity, maxDepth, maxBins)

// Evaluate model on test instances and compute test error
val labelsAndPredictions = testData.map { point =>
  val prediction = model.predict(point.features)
  (point.label, prediction)
}
val testMSE = labelsAndPredictions.map{ case (v, p) => math.pow(v - p, 2) }.mean()
println("Test Mean Squared Error = " + testMSE)
println("Learned regression tree model:\n" + model.toDebugString)

// Save and load model
model.save(sc, "target/tmp/myDecisionTreeRegressionModel")
val sameModel = DecisionTreeModel.load(sc, "target/tmp/myDecisionTreeRegressionModel")

The code correctly displays the model's MSE and learned tree model. However, I'm stuck in figuring out how to take the sameModel and use it to evaluate new data. Like, if the LibSVM file that I used to train the model looks like this:

0 1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:0 13:0 14:0 15:9 16:19
0 1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:1 13:0 14:0 15:9 16:12
0 1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:0 13:0 14:0 15:6 16:7

How do I feed the trained model something like this, and have it predict the label?

1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:0 13:0 14:0 15:9 16:19
1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:1 13:0 14:0 15:9 16:12
1:1.0 2:0.0 3:0.0 4:0.0 5:0.0 6:0.0 7:0.0 8:0.0 9:0.0 10:0.0 11:0.0 12:0 13:0 14:0 15:6 16:7

EDIT (8/31/2017 3:56 PM, Eastern)

Per the below suggestions, I'm trying the predict function, but it doesn't look like the code is quite right:

val new_data = MLUtils.loadLibSVMFile(sc, "hdfs://.../new_data/*")

val labelsAndPredictions = new_data.map { point =>
  val prediction = sameModel.predict(point.features)
  (point.label, prediction)
}

labelsAndPredictions.take(10)

If I run this with a LibSVM file containing '1' values as the label (I'm testing with ten new rows in the file), then they all come back as '1.0' in the labelsAndPredictions.take(10) command. If I give it a '0' value, then they all come back as '0.0', so it doesn't seem like anything's being predicted properly.

Answer 1

The load method should return a model. Then call predict with either a RDD[Vector] or a single Vector.

Answer 2

1. load raw data (as you did above, a similar LibSVM file)
2. provide information about categorical features
3. for each point in above data make predictions by calling: savedModel.predict(point.features)

Answer 3

You can load ML model from the disk via Pipeline :

import org.apache.spark.ml._
val pipeline = Pipeline.read.load("sample-pipeline")

scala> val stageCount = pipeline.getStages.size
stageCount: Int = 0

val pipelineModel = PipelineModel.read.load("sample-model")

scala> pipelineModel.stages

After obtaining pipeline use can make evaluation on the dataset:

val model = pipeline.fit(dataset)
val predictions = model.transform(dataset)

And you must use proper Evaluator eg RegressionEvaluator . Evaluator works on datasets with predictions:

import org.apache.spark.ml.evaluation.RegressionEvaluator
val regEval = new RegressionEvaluator
println(regEval.explainParams)
regEval.evaluate(predictions)

UPD If you have deal with hdfs you can easily load/save model:

One way to save a model to HDFS is as following:

// persist model to HDFS
sc.parallelize(Seq(model), 1).saveAsObjectFile("hdfs:///user/root/sample-model")

Saved model can then be loaded as:

val linRegModel = sc.objectFile[LinearRegressionModel]("hdfs:///user/root/sample-model").first()
linRegModel.predict(Vectors.dense(11.0, 2.0, 2.0, 1.0, 2200.0))

or like in example above, but instead local file hdfs :

PipelineModel.read.load("hdfs:///user/root/sample-model")

Answer 4

Ship the file with hdfs to a directory where all nodes can see it in the cluster. In your code load it and predict.