[英]Writing the model output to a text file spark scala
I fitted following logistic regression model using spark
MLlib
我使用spark
MLlib
val df = spark.read.option("header","true").option("inferSchema","true").csv("car_milage-6f50d.csv")
val hasher = new FeatureHasher().setInputCols(Array("mpg","displacement","hp","torque")).setOutputCol("features")
val transformed = hasher.transform(df)
val Array(training, test) = transformed.randomSplit(Array(0.8, 0.2))
val lr = new LogisticRegression()
.setFeaturesCol("features")
.setLabelCol("automatic")
.setMaxIter(20)
val paramGrid = new ParamGridBuilder()
.addGrid(lr.regParam, Array(0.1,0.3))
.addGrid(lr.elasticNetParam, Array(0.9,1))
.build()
val cv = new CrossValidator()
.setEstimator(lr)
.setEvaluator(new BinaryClassificationEvaluator())
.setEstimatorParamMaps(paramGrid)
.setNumFolds(10)
.setParallelism(2)
val model = cv.fit(training)
val results = model.transform(test).select("features", "automatic", "prediction")
val predictionAndLabels = results.select("prediction","label").as[(Double, Double)].rdd
At the end i obtained these model evaluation metrics最后我得到了这些 model 评估指标
val mMetrics = new MulticlassMetrics(predictionAndLabels)
mMetrics.confusionMatrix
mMetrics.labels
mMetrics.accuracy
As the file step I need to write these evaluation metrics ( mMetrics
) into a file (can be a text file of a csv file).作为文件步骤,我需要将这些评估指标( mMetrics
)写入文件(可以是 csv 文件的文本文件)。 Can anyone help me how to do that?谁能帮我怎么做?
I just tried and i couldn't find any write method which associated with these values.我刚刚尝试过,但找不到与这些值相关的任何写入方法。
Thank you谢谢
From looking at MultiClassMetrics 's method summary I think you should be able to do it this way:通过查看MultiClassMetrics的方法摘要,我认为您应该可以这样做:
val confusionMatrixOutput = mMetrics.confusionMatrix.toArray
val confusionMatrixOutputFinal = spark.parallelize(confusionMatrixOutput)
confusionMatrixOutputFinal.coalesce(1).saveAsTextFile("C:/confusionMatrixOutput.txt")
You should be able to do the same with mMetrics.labels:你应该能够对 mMetrics.labels 做同样的事情:
val labelsOutput = mMetrics.labels
val labelsOutputFinal = spark.parallelize(labelsOutput)
labelsOutputFinal.coalesce(1).saveAsTextFile("C:/labelsOutput.txt")
And accuracy should just be a double so you can just easily print this:准确度应该是两倍,所以你可以轻松地打印出来:
val accuracy = mMetrics.accuracy
println("Summary Statistics")
println(s"Accuracy = $accuracy")
You should be able to write all the statistics, for your logistic regression model, out to a single file like this:您应该能够将逻辑回归 model 的所有统计信息写入一个文件,如下所示:
import java.io._
object MulticlassMetricsOutputWriter {
def main(args:Array[String]) {
// All your other code can be added here
val mMetrics = new MulticlassMetrics(predictionAndLabels)
val labels = mMetrics.labels
// Create new file and passing reference of file to the printWriter
val pw = new PrintWriter(new File("C:/mllib_lr_output.txt"))
// Confusion Matrix
val confusionMatrixOutput = mMetrics.confusionMatrix.toArray
val confusionMatrixOutputFinal = spark.parallelize(confusionMatrixOutput)
pw.write(s"ConfusionMatrix:\n$confusionMatrixOutputFinal")
// Labels
val labelsOutput = mMetrics.labels
val labelsOutputFinal = spark.parallelize(labelsOutput)
pw.write(s"labels:\n$labelsOutputFinal")
// False positive rate by label
labels.foreach { l =>
pw.write(s"FPR($l) = " + mMetrics.falsePositiveRate(l) + "\n")
}
// True positive rate by label
labels.foreach { l =>
pw.write(s"TPR($l) = " + mMetrics.truePositiveRate(l) + "\n")
}
// F-measure by label
labels.foreach { l =>
pw.write(s"F1-Score($l) = " + mMetrics.fMeasure(l) + "\n")
}
// Precision by label
labels.foreach { l =>
pw.write(s"Precision($l) = " + mMetrics.precision(l) + "\n")
}
// Recall by label
labels.foreach { l =>
pw.write(s"Recall($l) = " + mMetrics.recall(l) + "\n")
}
val accuracy = mMetrics.accuracy
val weightedFalsePositiveRate = mMetrics.weightedFalsePositiveRate
val weightedFMeasure = mMetrics.weightedFMeasure
val weightedPrecision = mMetrics.weightedPrecision
val weightedRecall = mMetrics.weightedRecall
val weightedTruePositiveRate = mMetrics.weightedTruePositiveRate
pw.write("Summary Statistics" + "\n")
pw.write(s"Accuracy = $accuracy" + "\n")
pw.write(s"weightedFalsePositiveRate = $weightedFalsePositiveRate" + "\n")
pw.write(s"weightedFMeasure = $weightedFMeasure" + "\n")
pw.write(s"weightedPrecision = $weightedPrecision" + "\n")
pw.write(s"weightedRecall = $weightedRecall" + "\n")
pw.write(s"weightedTruePositiveRate = $weightedTruePositiveRate" + "\n")
// Closing the printWriter connection
pw.close
}
}
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.