在apache spark中選擇不同的（在一列上）非空（在所有其他列上）來自Dataframe的值

Question

我在Dataframe下面：

+---+--------+---------+-------+
|age|children|education| income|
+---+--------+---------+-------+
| 50|       2|     null|   null|
| 34|       4|     null|   null|
| 34|    null|     true|60000.0|
| 32|    null|    false|35000.0|
+---+--------+---------+-------+

我想要輸出如下所示：

+---+--------+---------+-------+
|age|children|education| income|
+---+--------+---------+-------+
| 50|       2|     null|   null|
| 34|       4|     true|60000.0|
| 32|    null|    false|35000.0|
+---+--------+---------+-------+

你可以看到年齡列包含34個重復所以我想合並34行的值（不是其他行的空值）

謝謝

Answer 1

如果首先在組中不為null，則可以使用“first”函數獲取：

val df = Seq(
  (50, Some(2), None, None),
  (34, Some(4), None, None),
  (34, None, Some(true), Some(60000.0)),
  (32, None, Some(false), Some(35000.0))
).toDF("age", "children", "education", "income")

val result = df
  .groupBy("age")
  .agg(
    first("children", ignoreNulls = true).alias("children"),
    first("education", ignoreNulls = true).alias("education"),
    first("income", ignoreNulls = true).alias("income")
  )
result.orderBy("age").show(false)

輸出：

+---+--------+---------+-------+
|age|children|education|income |
+---+--------+---------+-------+
|32 |null    |false    |35000.0|
|34 |4       |true     |60000.0|
|50 |2       |null     |null   |
+---+--------+---------+-------+