Using Spark ML's Logistic Regression model on MultiClass Classification giving error : Column prediction already exists

Question

I am using Spark ML's Logistic Regression model for classification problem having 100 categories (0-99). My columns in dataset are - "_c0,_c1,_c2,_c3,_c4,_c5" where _c5 is a target variable and rest are the features. My code is following :

import org.apache.spark.ml.feature.{StringIndexer, VectorAssembler}
import org.apache.spark.ml.classification.LogisticRegression
import org.apache.spark.ml.classification.OneVsRest
val _c0Indexer = new StringIndexer().setInputCol("_c0").setOutputCol("_c0Index")
val _c1Indexer = new StringIndexer().setInputCol("_c1").setOutputCol("_c1Index")
val _c2Indexer = new StringIndexer().setInputCol("_c2").setOutputCol("_c2Index")
val _c3Indexer = new StringIndexer().setInputCol("_c3").setOutputCol("_c3Index")
val _c4Indexer = new StringIndexer().setInputCol("_c4").setOutputCol("_c4Index")
val _c5Indexer = new StringIndexer().setInputCol("_c5").setOutputCol("_c5Index")
val assembler = new VectorAssembler().setInputCols(Array("_c0Index", "_c1Index", "_c2Index", "_c3Index","_c4Index")).setOutputCol("features")
val lr = new LogisticRegression()
  .setMaxIter(10)
  .setRegParam(0.3)
  .setElasticNetParam(0.8).setLabelCol("_c5Index").setFeaturesCol("features")
val ovr = new OneVsRest().setClassifier(lr)
val pipeline = new Pipeline().setStages(Array(_c0Indexer, _c1Indexer, _c2Indexer, _c3Indexer, _c4Indexer,assembler, _c5Indexer, ovr,lr))
val model = pipeline.fit(data)
val predictions = model.transform(testdf)
println(predictions.select("features", "_c5Index", "probability","prediction").show(5))

But it is showing an error :

requirement failed: Column prediction already exists.

Can someone please guide why I am getting this error? Thanks in advance.

Answer 1

Try taking out the "lr" at the end of your pipeline. I think it's unnecessary since ovr uses lr.