限制 spark scala 中 collect_list 项目的数量

Question

I have code that states

db.groupBy("ids")
  .agg(collect_list("names") as "alias")
  .select("ids", "alias")

However, there are some rows where the alias would be only 1 entry large, while others could be 200.

How can I limit the ones with >30 entries to a maximum of 30 entries?

Answer 1

You can use the slice function to subset the array.

db.groupBy("ids")                       
  .agg(slice(collect_list("names"), 1, 30) as "alias")
  .select("ids", "alias")