Pyspark date format

Question

I need to convert a string into a date format, for this the string format I have is something like this:

2021-01-07 11:17:49.385820+00:00

and what I am doing is the following:

    format = "yyyy-MM-dd HH:mm:ss.SSSSSS+ZZ:ZZ"
    level_1_data_df = level_1_data_value_df\
        .select(
        "level_1_data.*"
        ).withColumn("time2", to_timestamp(col("time"), format))

but the result of the new field is null, any ideas?

Answer 1

You can simply do

.withColumn("time2", to_timestamp(col("time")))

or, equivalently,

.withColumn("time2", col("time").cast("timestamp"))

because your timestamp has a standard format. No need for specifying its format.