使用 Scala 和 Spark 将日期列转换为年龄

Question

I am trying to convert a Column of a Dataset to true Age. I am using Scala with Spark and my project is on IntelliJ.

This is the sample dataset

TotalCost|BirthDate|Gender|TotalChildren|ProductCategoryName
1000||Male|2|Technology
2000|1957-03-06||3|Beauty
3000|1959-03-06|Male||Car
4000|1953-03-06|Male|2|
5000|1957-03-06|Female|3|Beauty
6000|1959-03-06|Male|4|Car
7000|1957-03-06|Female|3|Beauty
8000|1959-03-06|Male|4|Car 

And this is the code of Scala

import org.apache.spark.sql.SparkSession

object DataFrameFromCSVFile2 {

def main(args:Array[String]):Unit= {

val spark: SparkSession = SparkSession.builder()
  .master("local[1]")
  .appName("SparkByExample")
  .getOrCreate()

val filePath="src/main/resources/demodata.txt"

val df = spark.read.options(Map("inferSchema"->"true","delimiter"->"|","header"->"true")).csv(filePath).select("Gender", "BirthDate", "TotalCost", "TotalChildren", "ProductCategoryName")

val df2 = df
  .filter("Gender is not null")
  .filter("BirthDate is not null")
  .filter("TotalChildren is not null")
  .filter("ProductCategoryName is not null")
df2.show()

So I am trying to convert the 1957-03-06 to an age like 61 in the Column

Any idea will help a lot

Thank you very much

Answer 1

You can use the built-in functions - months_between() or datediff(). Check this out

scala> val df = Seq("1957-03-06","1959-03-06").toDF("date")
df: org.apache.spark.sql.DataFrame = [date: string]

scala> df.show(false)
+----------+
|date      |
+----------+
|1957-03-06|
|1959-03-06|
+----------+

scala> df.withColumn("age",months_between(current_date,'date)/12).show
+----------+------------------+
|      date|               age|
+----------+------------------+
|1957-03-06|61.806451612500005|
|1959-03-06|59.806451612500005|
+----------+------------------+

scala> df.withColumn("age",datediff(current_date,'date)/365).show
+----------+-----------------+
|      date|              age|
+----------+-----------------+
|1957-03-06|61.85205479452055|
|1959-03-06|59.85205479452055|
+----------+-----------------+


scala>

Answer 2

Here's one way that uses the java.time API in an UDF along with Spark's built-in when/otherwise for null check:

val currentAge = udf{ (dob: java.sql.Date) =>
  import java.time.{LocalDate, Period}
  Period.between(dob.toLocalDate, LocalDate.now).getYears
}

df.withColumn("CurrentAge", when($"BirthDate".isNotNull, currentAge($"BirthDate"))).
  show(5)
// +------+-------------------+---------+-------------+-------------------+----------+
// |Gender|          BirthDate|TotalCost|TotalChildren|ProductCategoryName|CurrentAge|
// +------+-------------------+---------+-------------+-------------------+----------+
// |  Male|               null|     1000|            2|         Technology|      null|
// |  null|1957-03-06 00:00:00|     2000|            3|             Beauty|        61|
// |  Male|1959-03-06 00:00:00|     3000|         null|                Car|        59|
// |  Male|1953-03-06 00:00:00|     4000|            2|               null|        65|
// |Female|1957-03-06 00:00:00|     5000|            3|             Beauty|        61|
// +------+-------------------+---------+-------------+-------------------+----------+

Answer 3

You can use the Java Calendar library to get the current date in your timezone to calculate the age. you can use udf to do that. for example

import java.time.ZoneId
import java.util.Calendar

val data = Seq("1957-03-06","1959-03-06").toDF("date")

val ageudf = udf((inputDate:String)=>{

val format = new java.text.SimpleDateFormat("yyyy-MM-dd")
val birthDate = format.parse(inputDate).toInstant.atZone(ZoneId.systemDefault()).toLocalDate
val currentDate = Calendar.getInstance().getTime..toInstant.atZone(ZoneId.systemDefault()).toLocalDate
import java.time.Period
if((birthDate != null) && (currentDate != null)) Period.between(birthDate,currentDate).getYears
else 0
})

data.withColumn("age",ageUdf($"date")).show()

The output will be:

date|age
1957-03-06|61
1959-03-06|59