如何在 pyspark 中将字典条目从字符串转换为时间戳？

Question

I have a dictionary with two dates:

dateFilter = {"date1": "2021-05-14", "date2": "2021-05-10"}

In my dataframe I have two columns of type timestamp:

|date1 | date2|
———————————————
|timea | timeb|
|timec | timed|

I want to filter this table based on whether the entries are smaller than the time in the dictionary

Table = Table.filter(F.col(column) <= DateFilters[date1])

The column needs to stay type timestamp so in order to compare I would like to convert the dictionary entries from String to Timestamp. I've tried this:

DateFilters[date1] = DateFilters[date1].cast(T.TimestampType)

But this doesn't work.

Answer 1

You can create a literal timestamp column from the dictionary value:

import pyspark.sql.functions as F

Table = Table.filter(F.col(column) <= F.lit(DateFilters["date1"]).cast('timestamp'))