将列添加到具有汇总值的DataFrame中

Question

I have a DataFrame as follows

profDF
+---+------------+---------+------+
| ID|        Name|      Occ|Salary|
+---+------------+---------+------+
|  1|       James|Detective| 30000|
|  2|      Victor| Salesman| 50000|
|  3|       Doris|      CEO| 20000|
+---+------------+---------+------+

I want to add a new column that contains the difference between the maximum Salary and the Salary of each person.

+---+------------+---------+------+-------+
| ID|        Name|      Occ|Salary|DiffMax|
+---+------------+---------+------+-------+
|  1|       James|Detective| 30000|  20000|
|  2|      Victor| Salesman| 50000|      0|
|  3|       Doris|      CEO| 20000|  30000|
+---+------------+---------+------+-------+

One way to do this would be create another DF by doing a groupBy("ID") and max and then join this DF with persDF on "ID" but groupBy would not give me the max Salary of all the rows.

Another way would be to use withColumn("DiffMax", ...) . But I can't seem to be able to find out the second parameter to withColumn that would give me the desired result.

Could someone help me with this? I'm using Spark-1.6.0

Answer 1

This is one way of doing it. Find the max Salary and then using withColumn find the difference between existing Salary and this max Salary.

val maxSalary = profDF.agg(max(profDF("Salary"))).first().get(0)

profDF.withColumn("DiffMax", lit(maxSalary) - profDF("Salary")).show()

//output

+---+------+---------+------+-------+
| ID|  Name|      Occ|Salary|DiffMax|
+---+------+---------+------+-------+
|  1| James|Detective| 30000|20000.0|
|  2|Victor| Salesman| 50000|    0.0|
|  3| Doris|      CEO| 20000|30000.0|
+---+------+---------+------+-------+