尝试将字符串转换为 pyspark 数据帧中的 unix_timestamp 数据类型时出现 SparkUpgrade 异常

Question

I am using spark3.1.1 and trying to convert string-datatype to unix_timestamp datatype using the below code.

pattern = 'yyyy/MM/dd hh:mm:ss aa'
file_new = file1.withColumn('Incident_DateTime', unix_timestamp(file1['Incident_DateTime'], pattern).cast('timestamp'))  
file_new.select('Incident_DateTime').show(5, False)

Getting error on all actions -> select/display/show. PFB snapshot and help

org.apache.spark.SparkUpgradeException: You may get a different result due to the upgrading of Spark 3.0: Fail to recognize 'yyyy/MM/dd hh:mm:ss aa' pattern in the DateTimeFormatter. 1) You can set spark.sql.legacy.timeParserPolicy to LEGACY to restore the behavior before Spark 3.0. 2) You can form a valid datetime pattern with the guide from https://spark.apache.org/docs/latest/sql-ref-datetime-pattern.html

Answer 1

From the link in your question:

"am-pm: This outputs the am-pm-of-day. Pattern letter count must be 1.",

so the pattern should be:

'yyyy/MM/dd hh:mm:ss a'