Alternative of Select DATE_FORMAT(date, format) in Apache Spark

Question

I am using Apache-Spark SQL and Java to read from parquet file. The file contains a date column( M/d/yyyy ) and I want to change that to some other format( yyyy-dd-MM ). Something like Select DATE_FORMAT(date, format) that we can do in mysql.
Is there any similar method in Apache-Spark?

Answer 1

What you can do is parse the string using to_timestamp with your current schema and format it with the one you desire using date_format :

val df = Seq("1/1/2015", "02/10/2014", "4/30/2010", "03/7/2015").toDF("d")
df.select('d, date_format(to_timestamp('d, "MM/dd/yyyy"), "yyyy-dd-MM") as "new_d")
  .show
+----------+----------+
|         d|     new_d|
+----------+----------+
|  1/1/2015|2015-01-01|
|02/10/2014|2014-10-02|
| 4/30/2010|2010-30-04|
| 03/7/2015|2015-07-03|
+----------+----------+

Note that the parsing is pretty robust and supports single digit days and months.