Spark Scala连接数据框减去列值

Question

I have two dataframes each has 2 columns. I want to join them by their 1st column and subtract their 2nd columns. Here's what I have so far:

var x = df.select("a", "c")
          .groubBy("a")
          .count()
var y = df.select("b", "c")
          .groubBy("b")
          .count()
var z = x.join(y, x("a") === y("b"))

How do I perform a dataframe subtraction? Without the dataframe, I usually to mapValues{case ..=> ..}. Thanks

Answer 1

val x = df.groubBy("a")
          .agg(sum("c").as("c1"))
          .select("a", "c1")
val y = df.groubBy("b")
          .agg(sum("c").as("c2"))
          .select("b", "c2")
val z = x.join(y, $"a" === $"b")
         .select($"a", $"c1" - $"c2")