Spark Java Encoders.bean 无法转换为 Scala 定义的 class

Question

I have Java code to convert a JavaRDD to Dataset and save it to HDFS:

Dataset<User> userDataset = sqlContext.createDataset(userRdd.rdd(), Encoders.bean(User.class));
userDataset.write.json("some_path");

User class is defined in Scala language:

case class User(val name: Name, val address: Seq[Address]) extends Serializable

case class Name(firstName: String, lastName: Option[String])

case class Address(address: String)

Code complies and runs successfully, file is saved to HDFS, while User class in the output file has empty schema:

val users = spark.read.json("some_path")
users.count // 100,000 which is same as "userRdd"
users.printSchema // users: org.apache.spark.sql.DataFrame = []

Why Encoders.bean is not working in this case?

Answer 1

Encoders.bean does not support Scala case class, Encoders.product supports that. Encoders.product takes a TypeTag as parameter while initializing a TypeTag is not possible in Java. I created a Scala object to provide TypeTag :

import scala.reflect.runtime.universe._

object MyTypeTags {
  val UserTypeTag: TypeTag[User] = typeTag[User]
}

Then in Java code: Dataset<User> userDataset = sqlContext.createDataset(userRdd.rdd(), Encoders.product(MyTypeTags.UserTypeTag()));