PySpark删除列中的无效日期时间格式

Question

My date time field format is: 2016-10-15 00:00:00 after using infer schema while saving my data to a parquet file, I have a few rows that don't comply to this format.

How can I collectively remove them in PySpark?

It is causing me problems in my UDF's.

Answer 1

Assuming you're parsing the date column and rows with invalid dates are null, which is usually the case:

df.filter(col('date').isNotNull())

Alternatively, if your date is read as a string, you can parse it using unix_timestamp :

(
    df
    .select(unix_timestamp('date', 'yyyy-MM-dd HH:mm:ss').cast("timestamp").alias('date'))
    .filter(col('date').isNotNull())
)