Spark Scala：将Case类对象添加到数据框

Question

I have a simple empty dataframe created as

import org.apache.spark.sql.SparkSession

lazy val sess = SparkSession.builder.appName("myapp").enableHiveSupport().getOrCreate()
case class MyClass (id:String, name:String)
val resultDf = sess.emptyDataset[MyClass]

now, i just want to create a new object, and append it to the dataframe. How do I do that? I have tried many things but failed

val x = MyClass("123", "zxc")
resultDf.union(x)

<console>:39: error: type mismatch;
 found   : ValidSignals
 required: org.apache.spark.sql.Dataset[ValidSignals]
       resultDf.union(x)

How can I convert the object to something that can be appended to the dataframe?

Answer 1

union expects a Dataset[_] as an argument. You need

val emptyDS = sess.emptyDataset[MyClass] 
val resultDS = emptyDS.union(Seq(MyClass("123","abc")).toDS())
resultDS.collect() // Array(MyClass(123,abc))