使用保存的模型预测新数据时出现火花数据帧格式错误

Question

I am able to train a model and saved the model (Train.scala). Now i want to use that trained model to predict on new data (Predict.scala).

I create a new VectorAssembler in my Predict.scala to featurize the new data. Should I use the same VectorAssembler in the Train.scala for the Predict.scala file? Because I am seeing issues with feature data type after transformation.

For example: when i read in the trained model and try to predict on the new data that is featurized, i got this error:

type mismatch;
[error]  found   : org.apache.spark.sql.DataFrame
[error]     (which expands to)  org.apache.spark.sql.Dataset[org.apache.spark.sql.Row]
[error]  required: org.apache.spark.sql.Dataset[org.apache.spark.sql.Row] => org.apache.spark.sql.Dataset[?]
[error]     val predictions = model.transform(featureData)

Training code: Train.scala

    // assembler
    val assembler = new VectorAssembler()
    .setInputCols(feature_list)
    .setOutputCol("features")

    //read in train data
    val trainingData = spark
      .read
      .parquet(train_data_path)

    // generate training features
    val trainingFeatures = assembler.transform(trainingData)

    //define model
    val lightGBMClassifier = new LightGBMClassifier()
        .setLabelCol("label")
        .setFeaturesCol("features")
        .setIsUnbalance(true)
        .setMaxDepth(25)
        .setNumLeaves(31)
        .setNumIterations(100) 

    // fit model
    val lgbm = lightGBMClassifier.fit(trainingFeatures)

    //save model
    lgbm
      .write
      .overwrite()
      .save(my_model_s3_path)

Predict code: Predict.scala

val assembler = new VectorAssembler()
    .setInputCols(feature_list)
    .setOutputCol("features")

// load model
val model = spark.read.parquet(my_model_s3_path)

// load new data
val inputData = spark.read.parquet(new_data_path)

//Assembler to transform new data
val featureData = assembler.transform(inputData)

//predict on new data
val predictions = model.transform(featureData) ### <- got error here

Should i be using a different method to read in my trained model or transform my data?

Answer 1

"Should I use the same VectorAssembler in the Train.scala for the Predict.scala file?" Yes, however, I would strong recommend to use Pipelines .

// Train.scala
val pipeline = new Pipeline().setStages(Array(assembler, lightGBMClassifier))
val pipelineModel = pipeline.fit(trainingData)
pipelineModel.write.overwrite().save("/path/to/pipelineModel")

// Predict.scala
val pipelineModel = PipelineModel.load("/path/to/pipelineModel")
val predictions = pipelineModel.transform(inputData)

See if the issue goes away but simply using Pipelines, serializing/deserializing the model correctly, and structuring your code better. Also, make sure that trainingData and inputData both contain the same columns listed in feature_list .