如何向我的 DataFrame 添加新列，以便新列的值由 scala 中的其他一些 function 填充？

Question

myFunc(Row): String = {
    //process row
    //returns string
}
appendNewCol(inputDF : DataFrame) : DataFrame ={
    inputDF.withColumn("newcol",myFunc(Row))
    inputDF
}

But no new column got created in my case. My myFunc passes this row to a knowledgebasesession object and that returns a string after firing rules. Can I do it this way? If not, what is the right way? Thanks in advance.

I saw many StackOverflow solutions using expr() sqlfunc(col(udf(x)) and other techniques but here my newcol is not derived directly from existing column.

Answer 1

Dataframe:

import org.apache.spark.sql.Row
import org.apache.spark.sql.types.{StringType, StructField, StructType}

val myFunc = (r: Row) => {r.getAs[String]("col1") + "xyz"} // example transformation

val testDf = spark.sparkContext.parallelize(Seq(
      (1, "abc"), (2, "def"), (3, "ghi"))).toDF("id", "col1")

testDf.show
    
val rddRes = testDf
        .rdd
        .map{x => 
          val y = myFunc (x)
          Row.fromSeq (x.toSeq ++ Seq(y) )
        }

val newSchema = StructType(testDf.schema.fields ++ Array(StructField("col2", dataType =StringType, nullable =false)))

spark.sqlContext.createDataFrame(rddRes, newSchema).show

Results:

+---+----+
| id|col1|
+---+----+
|  1| abc|
|  2| def|
|  3| ghi|
+---+----+

+---+----+------+
| id|col1|  col2|
+---+----+------+
|  1| abc|abcxyz|
|  2| def|defxyz|
|  3| ghi|ghixyz|
+---+----+------+

With Dataset:

case class testData(id: Int, col1: String)
case class transformedData(id: Int, col1: String, col2: String)

val test: Dataset[testData] = List(testData(1, "abc"), testData(2, "def"), testData(3, "ghi")).toDS

val transformedData: Dataset[transformedData] = test
  .map { x: testData =>
     val newCol = x.col1 + "xyz"
     transformedData(x.id, x.col1, newCol)
  }

transformedData.show

As you can see datasets is more readable, plus provides strong type casting. Since I'm unaware of your spark version, providing both solutions here. However if you're using spark v>=1.6, you should look into Datasets. Playing with rdd is fun, but can quickly devolve into longer job runs and a host of other issues that you wont foresee