在 Spark scakla 中对多个列使用 groupBy 和 agg

Question

I have a DataFrame with 4 columns. I want to apply GroupBy on the basis of 2 columns and want to collect other columns as list. Example:- I have a DF like this

+---+-------+--------+-----------+
|id |fName  |lName   |dob        |
+---+-------+--------+-----------+
|1  |Akash  |Sethi   |23-05-1995 |
|2  |Kunal  |Kapoor  |14-10-1992 |
|3  |Rishabh|Verma   |11-08-1994 |
|2  |Sonu   |Mehrotra|14-10-1992 |
+---+-------+--------+-----------+

and I want my output like this:-

+---+-----------+-------+--------+--------------------+
|id |dob        |fname           |lName               |
+---+-----------+-------+--------+--------------------+
|1  |23-05-1995 |[Akash]         |[Sethi]             |
|2  |14-10-1992 |[Kunal, Sonu]   |[Kapoor, Mehrotra]  |
|3  |11-08-1994 |[Rishabh]       |[Verma]             |
+---+-----------+-------+--------+--------------------+

Answer 1

You can do something like this using agg

df.groupBy("id","dob").agg(collect_list(col("fname")),collect_list(col("lName")))