我收到错误消息说没有类型的隐式参数：编码器 []" in Spark

Question

I get an error saying No implicit argument of type:Encoder[Movies] can you please tell me where am I going wrong as I am new to spark.

I am trying to read a movies file and converting it to a data set having 1 'ID' column and 2nd 'names of the movie' column.

import org.apache.spark.sql.SparkSession

object Practice {
    def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName("dataFrameExample")
          .master("local")
          .getOrCreate()

    **case class Movies( ID:String, name:String)**

        val ds1 = spark.read
         .format("text")
         .option("header", "true") //first line in file has headers
         .load("C:\\SparkScala\\SparkScalaStudy\\movies").as[Movies]

        ds1.printSchema()
    }
}

Answer 1

您需要将case class Movies移出main函数，并在ds1之前添加import spark.implicits._

Answer 2

You can import the sparksession.implicits to solve the problem or you can write your own implicits in an object as follows:

import org.apache.spark.sql.{Encoder, Encoders}

object CustomImplicits {    
  implicit val movieEncoder: Encoder[Movies] = Encoders.product[Movies]
}

Then simply import the implicit in your main method:

import package.containinig.implicits.CustomImplicits._
import org.apache.spark.sql.SparkSession
object Practice {
    def main(args: Array[String]): Unit = {
        val spark = SparkSession
          .builder()
          .appName("dataFrameExample")
          .master("local")
          .getOrCreate()

        val ds1 = spark.read
         .format("text")
         .option("header", "true") //first line in file has headers
         .load("C:\\SparkScala\\SparkScalaStudy\\movies").as[Movies]

        ds1.printSchema()
    }
}

Using the Encoders, you can enforce the Schema on your dataset as it would raise appropriate errors if the Schema is violated.