使用 Java 类的 Spark Scala 数据集

Question

I am creating a Spark application using the Scala binding. But some of my model's (classes) are written in Java. When I try to create a Dataset based on Scala Case Class, it works fine and all the columns are visible when I do show() . But when I create a Dataset based on a Java Class all the columns are packed in a single column named value .

Scala Case Class Example:

case class Person(name: String, age: Int)

Execution:

sqlContext.createDataset(Seq(Person("abcd", 10))).show()

Output:

name | age

abcd | 10

Java Class Example:

class Person {
  public String name;
  public int age;
  public Person (String name, int age) {
    this.name = name;
    this.age = age;
  }
}

Execution:

sqlContext.createDataset(Seq(Person("abcd", 10))).show()

Output:

value

[01 00 63 6F 6D 2...]

Are we not suppose to use Java classes as models with Spark Scala app? How do we resolve this issue?

Answer 1

You can use Java classes to create Datasets but you need to explictly define bean for that class (works like that in java). In addition you need to define getter/setter methods to define bean and your class definition should have public keyword(spark complains about some compliation errors). Hope it works okay for you.

Class

public class Person {
  private String name;
  private int age;

  public Person (String name, int age) {
    this.name = name;
    this.age = age;
  }

  public String getName() {
    return name;
  }

  public void setName(String name) {
    this.name = name;
  }

  public int getAge() {
    return age;
  }

  public void setAge(int age) {
    this.age = age;
  }
}

Execution

implicit val personEncoder = Encoders.bean(classOf[Person])
sql.createDataset(Seq(new Person("abcd", 10))).show()

Result

+---+----+
|age|name|
+---+----+
| 10|abcd|
+---+----+