Spark 中的累積乘積

Question

我嘗試在 Spark Scala 中實現一個累積產品，但我真的不知道如何實現。 我有以下數據框：

Input data:
+--+--+--------+----+
|A |B | date   | val|
+--+--+--------+----+
|rr|gg|20171103| 2  |
|hh|jj|20171103| 3  |
|rr|gg|20171104| 4  |
|hh|jj|20171104| 5  |
|rr|gg|20171105| 6  |
|hh|jj|20171105| 7  |
+-------+------+----+

我想有以下輸出：

Output data:
+--+--+--------+-----+
|A |B | date   | val |
+--+--+--------+-----+
|rr|gg|20171105| 48  | // 2 * 4 * 6
|hh|jj|20171105| 105 | // 3 * 5 * 7
+-------+------+-----+

Answer 1

只要數字是嚴格的正數（也可以處理 0，如果存在，使用coalesce ）如您的示例中所示，最簡單的解決方案是計算對數之和並取指數：

import org.apache.spark.sql.functions.{exp, log, max, sum}

val df = Seq(
  ("rr", "gg", "20171103", 2), ("hh", "jj", "20171103", 3), 
  ("rr", "gg", "20171104", 4), ("hh", "jj", "20171104", 5), 
  ("rr", "gg", "20171105", 6), ("hh", "jj", "20171105", 7)
).toDF("A", "B", "date", "val")

val result = df
  .groupBy("A", "B")
  .agg(
    max($"date").as("date"), 
    exp(sum(log($"val"))).as("val"))

由於這使用 FP 算術，因此結果將不准確：

result.show

+---+---+--------+------------------+
|  A|  B|    date|               val|
+---+---+--------+------------------+
| hh| jj|20171105|104.99999999999997|
| rr| gg|20171105|47.999999999999986|
+---+---+--------+------------------+

但四舍五入后應該足以滿足大多數應用程序。

result.withColumn("val", round($"val")).show

+---+---+--------+-----+
|  A|  B|    date|  val|
+---+---+--------+-----+
| hh| jj|20171105|105.0|
| rr| gg|20171105| 48.0|
+---+---+--------+-----+

如果這還不夠，您可以定義UserDefinedAggregateFunction或Aggregator （如何在 Spark SQL 中定義和使用用戶定義的聚合函數？ ）或使用帶有reduceGroups函數式 API：

import scala.math.Ordering

case class Record(A: String, B: String, date: String, value: Long)

df.withColumnRenamed("val", "value").as[Record]
  .groupByKey(x => (x.A, x.B))
  .reduceGroups((x, y) => x.copy(
    date = Ordering[String].max(x.date, y.date),
    value = x.value * y.value))
  .toDF("key", "value")
  .select($"value.*")
  .show

+---+---+--------+-----+
|  A|  B|    date|value|
+---+---+--------+-----+
| hh| jj|20171105|  105|
| rr| gg|20171105|   48|
+---+---+--------+-----+

Answer 2

您可以使用 collect_list+UDF 或 UDAF 解決此問題。 UDAF 可能更有效，但由於本地聚合而更難實現。

如果您有這樣的數據框：

+---+---+
|key|val|
+---+---+
|  a|  1|
|  a|  2|
|  a|  3|
|  b|  4|
|  b|  5|
+---+---+

您可以調用 UDF ：

val prod = udf((vals:Seq[Int]) => vals.reduce(_ * _))

df
  .groupBy($"key")
  .agg(prod(collect_list($"val")).as("val"))
  .show()

+---+---+
|key|val|
+---+---+
|  b| 20|
|  a|  6|
+---+---+

Answer 3

從 Spark 2.4 開始，您還可以使用高階函數aggregate來計算：

import org.apache.spark.sql.functions.{expr, max}
val df = Seq(
  ("rr", "gg", "20171103", 2),
  ("hh", "jj", "20171103", 3),
  ("rr", "gg", "20171104", 4),
  ("hh", "jj", "20171104", 5),
  ("rr", "gg", "20171105", 6),
  ("hh", "jj", "20171105", 7)
).toDF("A", "B", "date", "val")

val result = df
  .groupBy("A", "B")
  .agg(
    max($"date").as("date"),
    expr("""
   aggregate(
     collect_list(val),
     cast(1 as bigint),
     (acc, x) -> acc * x)""").alias("val")
  )

Answer 4

火花 3.2+

product(e: Column): Column
聚合函數：返回一個組中所有數字元素的乘積。

斯卡拉

import spark.implicits._
var df = Seq(
    ("rr", "gg", 20171103, 2),
    ("hh", "jj", 20171103, 3),
    ("rr", "gg", 20171104, 4),
    ("hh", "jj", 20171104, 5),
    ("rr", "gg", 20171105, 6),
    ("hh", "jj", 20171105, 7)
).toDF("A", "B", "date", "val")

df = df.groupBy("A", "B").agg(max($"date").as("date"), product($"val").as("val"))
df.show(false)
// +---+---+--------+-----+
// |A  |B  |date    |val  |
// +---+---+--------+-----+
// |hh |jj |20171105|105.0|
// |rr |gg |20171105|48.0 |
// +---+---+--------+-----+

火花

from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.getOrCreate()
data = [('rr', 'gg', 20171103, 2),
        ('hh', 'jj', 20171103, 3),
        ('rr', 'gg', 20171104, 4),
        ('hh', 'jj', 20171104, 5),
        ('rr', 'gg', 20171105, 6),
        ('hh', 'jj', 20171105, 7)]
df = spark.createDataFrame(data, ['A', 'B', 'date', 'val'])

df = df.groupBy('A', 'B').agg(F.max('date').alias('date'), F.product('val').alias('val'))
df.show()
#+---+---+--------+-----+
#|  A|  B|    date|  val|
#+---+---+--------+-----+
#| hh| jj|20171105|105.0|
#| rr| gg|20171105| 48.0|
#+---+---+--------+-----+