Spark scala dataframe 获取每行的值并分配给变量

Question

I have a dataframe like below:

val df=spark.sql("select * from table")

row1|row2|row3

A1,B1,C1

A2,B2,C2

A3,B3,C3

i want to iterate for loop to get values like this:

val value1="A1"

val value2="B1"

val value3="C1"

function(value1,value2,value3)

Please help me.

emphasized text

Answer 1

You have 2 options:

• Solution 1- Your data is big, then you must stick with dataframes. So to apply a function on every row. We must define a UDF.

• Solution 2- Your data is small, then you can collect the data to the driver machine and then iterate with a map.

Example:

val df = Seq((1,2,3), (4,5,6)).toDF("a", "b", "c")
def sum(a: Int, b: Int, c: Int) = a+b+c

// Solution 1
import org.apache.spark.sql.Row
val myUDF = udf((r: Row) => sum(r.getAs[Int](0), r.getAs[Int](1), r.getAs[Int](2)))

df.select(myUDF(struct($"a", $"b", $"c")).as("sum")).show

//Solution 2
df.collect.map(r=> sum(r.getAs[Int](0), r.getAs[Int](1), r.getAs[Int](2))) 

Output for both cases:

+---+
|sum|
+---+
|  6|
| 15|
+---+

EDIT:

val myUDF = udf((r: Row) => {
  val value1 = r.getAs[Int](0)
  val value2 = r.getAs[Int](1)
  val value3 = r.getAs[Int](2)

  myFunction(value1, value2, value3)
})