更新 Scala 中的数据框列

Question

Given dataframe with columns A, B, and C, created with a "val x = ", I want to update a column like this:

x.withColumn("A", when ($"B" === "apple", "fruit").otherwise(col("C")))

This doesn't actually change x, which I believe is expected. Most people I think would create a new dataframe:

val y = x.withColumn("A", when ($"B" === "apple", "fruit").otherwise(col("B")))

And y has the update. But how do you change x with creating a new dataframe? I realize val x is immutable, but even when I declare "var x", it's the same behavior. It doesn't actually save the change.

Is that the Scala best practice, to always create a new DF?

Answer 1

As per Spark Architecture DataFrame is built on top of RDDs which are immutable in nature, Hence Dataframes are also immutable in nature.

The withColumn or any other operation for that matter, on DataFrames, will generate a new data frame instead of updating the existing data frame.

val y = x.withColumn("A", when ($"B" === "apple", "fruit").otherwise(col("B")))

You are just storing the result in val y .