[英]Spark Scala Conditionally add to agg
Is it possible to add an aggregate conditionally in Spark Scala?是否可以在 Spark Scala 中有条件地添加聚合?
I would like to DRY out the following code by conditionally adding collect_set
我想通过有条件地添加collect_set
来干掉以下代码
Example:例子:
val aggDf = if (addId) groups.agg(
count(lit(1)).as("Count"),
percentile_approx($"waitTime",lit(0.5), lit(10000)),
collect_set("Id").as("Ids")
)
else groups.agg(
count(lit(1)).as("Count"),
percentile_approx($"waitTime",lit(0.5), lit(10000))
)
Maybe the is a better way of writing the whole code.也许这是编写整个代码的更好方法。
Thanks.谢谢。
You can store the aggreate columns in a sequence and alter the sequence as required:您可以按顺序存储聚合列并根据需要更改顺序:
var aggCols = Seq(count(lit(1)).as("Count"),
percentile_approx($"waitTime",lit(0.5), lit(10000)))
if(addId) aggCols = aggCols :+ collect_set("Id").as("Ids")
val aggDf = groups.agg(aggCols.head, aggCols.tail:_*)
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.