如何在Spark Scala中使用数据集设置数组类型

Question

I have a source data like this:

{A:123,B:"Hello world",C:[{D:123,E:"Spark"}]}

And i have a Object:

case class TestClass (A:Int;B:String;C:???)
val obj:Dataset[TestClass] = df.as[TestClass]

How should I define the type of C?

Answer 1

One option

case class Nested(D: Long, E: String)
case class TestClass (A: Long, B:String, C: Seq[Nested])

Usage:

spark.read.json(sc.parallelize(
  Seq("""{"A": 123, "B": "Hello world", "C": [{"D": 123, "E": "Spark"}]}"""
))).as[TestClass].show

+---+-----------+-------------+
|  A|          B|            C|
+---+-----------+-------------+
|123|Hello world|[[123,Spark]]|
+---+-----------+-------------+