Java / Spark-按加权平均聚合分组

Question

data :

id | sector     | balance
---------------------------
1  | restaurant | 20000
2  | restaurant | 20000
3  | auto       | 10000
4  | auto       | 10000
5  | auto       | 10000

i am looking to load this into spark as a df and calculate group by balance sums, but i also have to calculate the balace% against total balance (sum(balance) for all ids)

how can I accomplish this ?

Answer 1

To get the % against total you could use the DoubleRDDFunctions:

val totalBalance = data.map(_._3.toDouble).sum()

val percentageRow = data.map(d => d._3 * 100 / totalBalance)

val percentageGroup = data.map(d => (d._2, d._3))
         .reduceByKey((x,y) => x+y).mapValues(sumGroup => sumGroup * 100 / totalBalance)