字符串到日期时间的转换失败

Question

The conversion of the string to datetime is failing. The data in the dataframe has the following format: "2020-08-05T12:34:10.800046" .
I used pattern yyyy-MM-ddTHH:mm:ss.SSSSSS

config_df.withColumn(
    "modifiedDate",
    F.to_timestamp(config_df["modifiedDate"], "yyyy-MM-dd'T'HH:mm:ss.SSSSSS"),
).show()

+------------+
|modifiedDate|
+------------+
|        null|
+------------+

The execution works without problem but all values in the updated column are NULL . Which format should I use?

Answer 1

According to this post , SSS is for milliseconds. Therefore, it matches the first 3 digits 800 in your 800046 , no matter how many S you add.

I couldn't find any pattern that match your date, so you first need to update your string to keep only 3 digits at the end. With a regex for example

a = [
    ("2020-08-05T12:34:10.800123",),
]
b = ["modifiedDate"]
df = spark.createDataFrame(a, b)

df.withColumn(
    "modifiedDate",
    F.to_timestamp(
        F.regexp_extract(
            "modifiedDate", r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}", 0
        ),
        "yyyy-MM-dd'T'HH:mm:ss.SSS",
    ),
).show()


+-------------------+
|       modifiedDate|
+-------------------+
|2020-08-05 12:34:10|
+-------------------+