如何将“ 01MAR1978：00：00：00”字符串格式的日期转换为SparkR中的日期格式？

Question

I have dates in the following formats:

1. 08MAR1978:00:00:00
2. 10FEB1973:00:00:00
3. 15AUG1982:00:00:00

I would like to convert them to:

1. 1978-03-08
2. 1973-02-10
3. 1982-09-15

I have tried the following in SparkR:

period_uts <- unix_timestamp(all.new$DATE_OF_BIRTH, '%d%b%Y:%H:%M:%S')
period_ts <- cast(period_uts, 'timestamp')
period_dt <- cast(period_ts, 'date')
df <- withColumn(all.new, 'p_dt', period_dt)    

But when I do this, all the dates get changed into "NA".

Can anyone please provide some insights on how I can convert dates in %d%B%Y:%H:%M:%S format to dates in SparkR?

Thanks!

Answer 1

I don't think you need SparkR to solve this question.

What you have:

DoB <- c("08MAR1978:00:00:00", "10FEB1973:00:00:00", "15AUG1982:00:00:00")

If you want to get 1978-03-08 etc. you could just use as.Date in combination with the date format you already found yourself:

as.Date(DoB, format="%d%B%Y:%H:%M:%S")
# [1] "1978-03-08" "1973-02-10" "1982-08-15"

as.Date will ensure that R knows how to interpret your string as a date.

Note, however, that in general the way dates are displayed to you (ie 1978-03-08) actually don't really matter. The reason is that 'under the hood', R understands your date now, so all date-related operations will be performed appropriately.

Answer 2

I figured out how to do it:

all.new = all.new %>% withColumn("Date_of_Birth_Fixed", to_date(.$DATE_OF_BIRTH,  "ddMMMyyyy"))

This works in Spark 2.2.x