Apache Spark Scala with Play-json Validation

Question

java.lang.UnsupportedOperationException: Schema for type [trait object] is not supported

trait Container {
  def aa: String
  def bb: Int
}

case class First(aa: String, bb: Int) extends Container
case class Second(aa: String, bb: Int) extends Container

implicit val aaContainerFormat: Format[First] = Json.format[First]

implicit val bbContainerFormat: Format[Second] = Json.format[Second]

implicit def nodeContainerReads: Reads[Container] =
  try {
    Json.format[First].map(x => x: Container) or
    Json.format[Second].map(x => x: Container)
  } catch {
    case e: Exception => Reads {
      case _ => JsError(JsonValidationError("Cannot De-serialize value."))
    }
  }

implicit def nodeContainerWrites = new Writes[Container] {
  override def writes(node: Container): JsValue = node match {
    case a: First => Json.toJson(a)
    case b: Second => Json.toJson(b)
    case _ => Json.obj("error" -> "wrong Json")
  }
}

// Example Usage....
val spark: SparkSession = SparkSession.builder.appName("Unit Test").getOrCreate()
val js: Container = First("unit", "test")

spark.createDataFrame(Seq(js))

I expect the output of Datasets of [Container Object] but the actual output is java.lang.UnsupportedOperationException: Schema for type Container is not supported.

Answer 1

Spark does not use typeclasses from Play JSON to convert Scala types into Spark SQL types. Instead you need to look at Spark Encoders which form the basis of conversions for Scala types to Spark types. If you have the Spark Session in scope, you can use import sparkSession.implicits._ so it will automatically create encoders for your case classes. I believe that Spark doesn't support sum-types out of the box so you would need to implement your own Encoder to somehow model that in Spark in an ad-hoc fashion. Please read here for more information if you want to encode sum types in Spark