[英]How can spark infer dataset automatically after join of two datasets?
spark 是否可以自动推断模式并将 Dataframe 转换为数据集,而程序员不必为每个连接创建一个案例 class?
import spark.implicits._
case class DfLeftClass(
id: Long,
name: String,
age: Int
)
val dfLeft = Seq(
(1,"Tim",30),
(2,"John",15),
(3,"Pens",20)
).toDF("id","name", "age").as[DfLeftClass]
case class DfRightClass(
id: Long,
name: String,
age: Int
hobby: String
)
val dfRight = Seq(
(1,"Tim",30,"Swimming"),
(2,"John",15,"Reading"),
(3,"Pens",20,"Programming")
).toDF("id","name", "age", "hobby").as[DfRightClass]
val joined: DataFrame = dfLeft.join(dfRight) // this results in DataFrame instead of a Dataset
要留在数据集 API 中,您可以使用joinWith 。 这个 function 返回包含连接两侧的元组数据集:
val joined: Dataset[(DfLeftClass, DfRightClass)] = dfLeft.joinWith(dfRight,
dfLeft.col("id").eqNullSafe(dfRight.col("id")))
结果:
+-------------+--------------------------+
|_1 |_2 |
+-------------+--------------------------+
|{1, Tim, 30} |{1, Tim, 30, Swimming} |
|{2, John, 15}|{2, John, 15, Reading} |
|{3, Pens, 20}|{3, Pens, 20, Programming}|
+-------------+--------------------------+
从这里您可以继续使用元组,也可以将元组 map 转换为第三种情况 class。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.