如何在 PySpark 中复制 Pandas 的 between_time function

Question

I want to replicate the between_time function of Pandas in PySpark. Is it possible since in Spark the dataframe is distributed and there is no indexing based on datetime?

i = pd.date_range('2018-04-09', periods=4, freq='1D20min')
ts = pd.DataFrame({'A': [1, 2, 3, 4]}, index=i)
ts.between_time('0:45', '0:15')

Is something similar possible in PySpark?

pandas.between_time - API

Answer 1

If you have a timestamp column, say ts , in a Spark dataframe, then for your case above, you can just use

import pyspark.sql.functions as F

df2 = df.filter(F.hour(F.col('ts')).between(0,0) & F.minute(F.col('ts')).between(15,45))