Spark根据row_number的最大值和另一列的字符串值创建一个新的字符串列

Question

Suppose we have

id, hit, item, row_number
1234, 1, item1, 1
1234, 2, item2, 2
2345, 2, item1, 1
2345, 2, item2, 2
2345, 4, item3, 3

where row_number was created from windows function partitioned by id on hit in ascending order.

Now, I would like to create a new column max_hit_item which contains the name of the item with the highest row_number per user.

So in our example, it would return,

id, hit, item, row_number, max_hit_item
1234, 1, item1, 1, item2
1234, 2, item2, 2, item2
2345, 2, item1, 1, item3
2345, 2, item2, 2, item3
2345, 4, item3, 3, item3

I'm thinking since I do not want to drop any rows, I will have to use windows function. Is there a unique way of using windows function to achieve this? Ideally, I would like to not use join but any solutions are welcome.

Answer 1

Using window function first .

import org.apache.spark.sql.expressions.Window
val w = Window.partitionBy("id").orderBy(desc("hit"))
val result = df.withColumn("max_hit_item", first("item").over(w))