如何进行 groupby 排名并将其作为列添加到 spark scala 中的现有 dataframe？

Question

Currently this is what I'm doing:

  val new_df= old_df.groupBy("column1").count().withColumnRenamed("count","column1_count")

  val new_df_rankings = new_df.withColumn(
    "column1_count_rank",
    dense_rank()
      .over(
        Window.orderBy($"column1_count".desc))).select("column1_count","column1_count_rank")

But really all I'm looking to do is add a column to the original df (old_df) called "column1_count_rank" without going through all these intermediate steps and merging back.

Is there a way to do this?

Thanks and have a great day!

Answer 1

As you apply aggregation, there will be a calculative result, it will create new dataframe. Can you give some sample input and output example

old_df.groupBy("column1").agg(count("*").alias("column1_count")).withColumn("column1_count_rank",dense_rank().over(Window.orderBy($"column1_count".desc))).select("column1_count","column1_count_rank")