使用Apache Spark和Java對DataFrame進行分組和聚合？

Question

我有一個DataFrame，在Spark中加載了以下模式：

email，first_name，last_name，order_id

如何通過電子郵件對其進行分組，計算每個組中的記錄並使用此模式返回DataFrame：

email，first_name，last_name，order_count

Answer 1

這是在Scala中執行此操作的方法：

val df = sc.parallelize(Seq(("a","b","c",1),("a","b","c",2),("x","xb","xc",3),("y","yb","yc",4),("x","xb","xc",5))).toDF("email","first_name","last_name","order_id")

df.registerTempTable("df")

sqlContext.sql("select * from (select email, count(*) as order_count from df group by email ) d1 join df d2 on d1.email = d2.email")

在Java中 ，考慮到您已經創建了DataFrame，它實際上是相同的代碼：

DataFrame results = sqlContext.sql("select * from (select email, count(*) as order_count from df group by email ) d1 join df d2 on d1.email = d2.email");

盡管如此，甚至認為這是一個直接的解決方案，但我認為這是一個不好的做法，因為你的代碼很難維護和發展。 更清潔的解決方案是：

DataFrame email_count = df.groupBy("email").count();
DataFrame results2 = email_count.join(df, email_count.col("email").equalTo(df.col("email"))).drop(df.col("email"));