Spark DataFrame convert milliseconds timestamp column in string format to human readable time with milliseconds

Question

I have a Spark DataFrame with a timestamp column in milliseconds since the epoche. The column is a string . I now want to transform the column to a readable human time but keep the milliseconds. For example:

1614088453671 -> 23-2-2021 13:54:13.671

Every example i found transforms the timestamp to a normal human readable time without milliseconds.

What i have:

+------------------+
|epoch_time_seconds|
+------------------+
|1614088453671     |
+------------------+

What i want to reach:

+------------------+------------------------+
|epoch_time_seconds|human_date              |
+------------------+------------------------+
|1614088453671     |23-02-2021 13:54:13.671 |
+------------------+------------------------+

Answer 1

The time before the milliseconds can be obtained using date_format from_unixtime , while the milliseconds can be obtained using a modulo. Combine them using format_string .

val df2 = df.withColumn(
    "human_date",
    format_string(
        "%s.%s",
        date_format(
            from_unixtime(col("epoch_time_seconds")/1000),
            "dd-MM-yyyy HH:mm:ss"
        ),
        col("epoch_time_seconds") % 1000
    )
)

df2.show(false)
+------------------+-----------------------+
|epoch_time_seconds|human_date             |
+------------------+-----------------------+
|1614088453671     |23-02-2021 13:54:13.671|
+------------------+-----------------------+