如何使用一些类似的值和多个键/ scala连接数据帧

Question

I have problems to get following table. The first two tables are my source tables which i would like to join. the third table is how i would like to have it.

I tried it with an outer join and used the keys "ID" and "date" but the result is not the same like in this example. The problem is, that some def_ values in each table have the same date and i would like to get them in the same row.

I used following join:

val df_result = df_1.join(df_2, Seq("ID", "date"), "outer")

df
+----+-----+-----------+
|ID  |def_a| date      |
+----+-----+-----------+
|  01|    1| 2019-01-31|
|  02|    1| 2019-12-31| 
|  03|    1| 2019-11-30|
|  01|    1| 2019-10-31|

df
+----+-----+-----+-----------+
|ID  |def_b|def_c|date       |
+----+-----+-----+-----------+
|  01|    1|    0| 2017-01-31| 
|  02|    1|    1| 2019-12-31| 
|  03|    1|    1| 2018-11-30| 
|  03|    0|    1| 2019-11-30| 
|  01|    1|    1| 2018-09-30|
|  02|    1|    1| 2018-08-31|
|  01|    1|    1| 2018-07-31|

result
+----+-----+-----+-----+-----------+
|ID  |def_a|def_b|deb_c|date       |
+----+-----+-----+-----+-----------+
|  01|    1|    0|    0| 2019-01-31| 
|  02|    1|    1|    1| 2019-12-31| 
|  03|    1|    0|    1| 2019-11-30| 
|  01|    1|    0|    0| 2019-10-31| 
|  01|    0|    1|    0| 2017-01-31| 
|  03|    0|    1|    1| 2018-11-30| 
|  01|    0|    1|    1| 2018-09-30| 
|  02|    0|    1|    1| 2018-08-31|
|  01|    0|    1|    1| 2018-07-31|

I would be grateful for any help.

Answer 1

Hope the following code would be helpful —

df_result
    .groupBy("ID", "date")
    .agg(
        max("a"),
        max("b"),
        max("c")
        )