如何使用Scala / Spark 2.2将列添加到现有DataFrame并使用window函数在新列中添加特定行

Question

Eg: I would like to add the quantity sold by the date.

Date       Quantity
11/4/2017    20 
11/4/2017    23 
11/4/2017    12 
11/5/2017    18
11/5/2017    12

Output with the new Column:

Date        Quantity, New_Column
11/4/2017      20        55
11/4/2017      23        55
11/4/2017      12        55
11/5/2017      18        30
11/5/2017      12        30

Answer 1

Simply use sum as a window function by specifying a WindowSpec :

import org.apache.spark.sql.expressions.Window

df.withColumn("New_Column", sum("Quantity").over(Window.partitionBy("Date"))).show
+---------+--------+----------+
|     Date|Quantity|New_Column|
+---------+--------+----------+
|11/5/2017|      18|        30|
|11/5/2017|      12|        30|
|11/4/2017|      20|        55|
|11/4/2017|      23|        55|
|11/4/2017|      12|        55|
+---------+--------+----------+