spark convert datetime to timestamp

Question

I have a column in pyspark dataframe which is in format 2021-10-28T22:19:03.0030059Z(string datatype). How to convert this into a timestamp datatype in pyspark? I'm using the code snippet below but this returns nulls as its unable to convert it. Can someone please recommend on how to convert this?

df3.select(to_timestamp(df.DateTime, 'yyyy-MM-ddHH:mm:ss:SSS').alias('dt'),col('DateTime')).show()

Answer 1

You have to escape (put it in '') T and Z:

import pyspark.sql.functions as F

df = spark.createDataFrame([{"DateTime": "2021-10-28T22:19:03.0030059Z"}])

df.select(F.to_timestamp(df.DateTime, "yyyy-MM-dd'T'HH:mm:ss.SSSSSSS'Z'").alias('dt'),F.col('DateTime')).show(truncate = False)`