在Spark Window函数中，为什么我们需要在末尾使用drop（）

Question

I'm new to Spark window functions. I am implementing few examples to learn more about it. Take a look at below example. It's using drop() with withColumn(). I searched a lot on spark docs as well but couldn't able to understand its significance.

//Get the top record in each subject with the highest fee
val wSpec = Window.partitionBy($"Subject").orderBy($"Fee".desc)
val dfTop = input.withColumn("rn", row_number.over(wSpec)).where($"rn"===1).drop("rn") //Note: 'input' has my data 
dfTop.show()

Can someone explain the significance of drop()? What if I do not use drop()?

Thanks.

Answer 1

Why we need to use drop() at the end

We don't. We do it to remove temporary objects, which no longer carries useful information.

What if I do not use drop()?

You'll have one more column, full of ones, nothing more, nothing less.

Answer 2

drop() is used to drop the column(s) which you don't want any more further, nothing much significance.

You can see it yourself simply by:

//Commenting drop()
val dfTop = input.withColumn("rn", row_number.over(wSpec)).where($"rn"===1) //.drop("rn") //Note: 'input' has my data 
dfTop.show()

dfTop.drop("rn").show()
//"rn" column is gone