Spark Scala：如何转换DF中的列

Question

I have a dataframe in Spark with many columns and a udf that I defined. I want the same dataframe back, except with one column transformed. Furthermore, my udf takes in a string and returns a timestamp. Is there an easy way to do this? I tried

val test = myDF.select("my_column").rdd.map(r => getTimestamp(r)) 

but this returns an RDD and just with the transformed column.

Answer 1

If you really need to use your function, I can suggest two options:

1) Using map / toDF:

import org.apache.spark.sql.Row
import sqlContext.implicits._

def getTimestamp: (String => java.sql.Timestamp) = // your function here

val test = myDF.select("my_column").rdd.map {
  case Row(string_val: String) => (string_val, getTimestamp(string_val))
}.toDF("my_column", "new_column")

2) Using UDFs ( UserDefinedFunction ):

import org.apache.spark.sql.functions._

def getTimestamp: (String => java.sql.Timestamp) = // your function here

val newCol = udf(getTimestamp).apply(col("my_column")) // creates the new column
val test = myDF.withColumn("new_column", newCol) // adds the new column to original DF

There's more detail about Spark SQL UDFs in this nice article by Bill Chambers .

---

Alternatively ,

If you just want to transform a StringType column into a TimestampType column you can use the unix_timestamp column function available since Spark SQL 1.5:

val test = myDF
  .withColumn("new_column", unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm").cast("timestamp"))

Note: For spark 1.5.x, it is necessary to multiply the result of unix_timestamp by 1000 before casting to timestamp (issue SPARK-11724 ). The resulting code would be:

val test = myDF
  .withColumn("new_column", (unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm") *1000L).cast("timestamp"))

Edit: Added udf option