使用动态列数更改数据框行值

Question

I have a dataframe (contains 10 columns) for which I want to change the value of a row (for the last column only). I have written following code for this:

 val newDF = spark.sqlContext.createDataFrame(WRADF.rdd.map(r=> {
      Row(r.get(0), r.get(1),
          r.get(2), r.get(3),
          r.get(4), r.get(5),
          r.get(6), r.get(7),
          r.get(8), decrementCounter(r))
     }), WRADF.schema) 

I want to change the value of a row for 10th column only (for which I wrote decrementCounter() function). But the above code only runs for dataframes with 10 columns. I don't know how to convert this code so that it can run for different dataframe (with different number of columns). Any help will be appreciated.

Answer 1

Don't do something like this. Define udf

import org.apache.spark.sql.functions.udf._

val decrementCounter = udf((x: T) => ...) // adjust types and content to your requirements

df.withColumn("someName", decrementCounter($"someColumn"))

Answer 2

I think UDF will be a better choice because it can be applied using the Column name itself.

For more on udf you can take a look here : https://docs.databricks.com/spark/latest/spark-sql/udf-scala.html

For your code just use this :

import org.apache.spark.sql.functions.udf._

val decrementCounterUDF = udf(decrementCounter _) 

df.withColumn("columnName", decrementCounterUDF($"columnName"))

What it will does is apply this decrementCounter function on each and every value of column columnName .

I hope this helps, cheers !