How to convert a string column (column which contains only time and not date ) to time_stamp in spark-scala?

Question

I need to convert the column which contains only time as string to a time stamp type or any other time function which is available in spark.

Below is the test Data frame which having "Time_eg" as string column,

Time_eg
12:49:09 AM
12:50:18 AM

Schema before it convert to the time,

Time_eg: string (nullable = true)

//Converting to time stamp
val transType= test.withColumn("Time_eg", test("Time_eg").cast("timestamp"))

Schema After converting to timestamp, the schema is

Time_eg: timestamp (nullable = true)

But the output of transType.show() gives null value for the "Time_eg" column.

Please let me know how to convert the column which contains only time as a string to time stamp in spark scala?

Much appreciate if anyone can help on this?

Thanks

Answer 1

You need to use a specific function to convert a string to a timestamp, and specify the format. Also, a timestamp in Spark represents a full date (with time of the day). If you do not provide the date, it will be set to 1970, Jan 1st, the begining of unix timestamps.

In your case, you can convert your strings as follows:

Seq("12:49:09 AM", "09:00:00 PM")
    .toDF("Time_eg")
    .select(to_timestamp('Time_eg, "hh:mm:ss aa") as "ts")
    .show
+-------------------+
|                 ts|
+-------------------+
|1970-01-01 00:49:09|
|1970-01-01 21:00:00|
+-------------------+