没有 TypeTag 可用于案例 class 使用 scala 3 和火花 3

Question

I have my code that runs a spark job with scala 3

@main def startDatasetJob(): Unit =
  val spark = SparkSession.builder()
    .appName("Datasets")
    .master("local[*]")
    .getOrCreate()

  case class CarRow(Name: String,
                 Miles_per_Gallon: Double,
                 Cylinders: Long,
                 Displacement: Double,
                 Horsepower: Long,
                 Weight_in_lbs: Long,
                 Acceleration: Double,
                 Year: Date,
                 Origin: String
                )

  implicit val carEncoder: Encoder[CarRow] = Encoders.product[CarRow]
  val carsDF = spark.read
    .format("json")
    .option("inferSchema", "true")
    .load("src/main/resources/data/cars.json")

  val carDS = carsDF.as[CarRow]

but get the error message

No TypeTag available for CarRow
  implicit val carEncoder: Encoder[CarRow] = Encoders.product[CarRow]

get a bit confused why the compiler seems unable to load the case class, if any one can help

Answer 1

Spark's encoder derivation relies on Scala 2.x mechanisms, some of them no longer exist.

Have you tried with spark-scala3 ? It provides the encoder derivation for Scala 3.

libraryDependencies += "io.github.vincenzobaz" %% "spark-scala3" % "0.1.3"

Then

import scala3encoders.given

(see the examples in the repo)