检索行列值作为其Scala类型而不是列

Question

What I'm trying to achieve is inferring values to certain DataFrame columns taking into account values of each individual row.

.withColumn("date", when(col("date").isNull, lit(new DateTime(col("timestamp").as[Long]).getYear)))

The problem is that I can't wrap my head around how to retrieve, for each of the Row objects, its value for the given column. I've seen other solutions but they either list the whole set of values for all of the rows, or just get the first value of them, which isn't what I'm trying to achieve.

Image an example DF like this...

 (year, val1, val2, val3, timestamp) (null, 10, 12, null, 123456789) (null, 11, 12, null, 234567897) 

And what I want to see after applying individual functions (for example, extracting year from timestamp) to each of the Rows is...

 (year, val1, val2, val3, timestamp) (2018 [using DateTime class], 10, 12, 1012, 123456789) (2018 [using DateTime class], 12, 12, 1212, 234567897) 

Is there any way of doing this?

Answer 1

Thats where UDFs come into play :

val udf_extractYear = udf((ts:Long) => new DateTime(ts).getYear)

then you can use this using eg

df
.withColumn("year", when(col("year").isNull, udf_extractYear(col("timestamp"))).otherwise(col("year")))
.show()

As you can see your timestamp column is automatically mapped to Long