PySpark - 为组创建特定的时间序列范围

Question

I have 1000 Tags that have a Timestamp and a Value . For each of the Tags the date range is '2020-01-01', however this is too much data for each Tag. I have a separate dataframe that has a Start and End for each of the Tags in the first dataframe.

I only need data from the date ranges mentioned above in the data with the 1000 Tags data. I also need the time series data in the desired dataframe padded 2 days prior to the Start and 1 day after the End dates.

df1 = spark.createDataFrame(
    [("Tag 1", "2020-05-01", 1), ("Tag 1000", "2021-02-01", 1),
        ("Tag 1", "2020-05-02", 2), ("Tag 1000", "2021-02-02", 2),
        ("Tag 1", "2020-05-03", 3), ("Tag 1000", "2021-02-03", 3),
        ("Tag 1", "2020-05-04", 4), ("Tag 1000", "2021-02-04", 4),
        ("Tag 1", "2020-05-05", 5), ("Tag 1000", "2021-02-05", 5),
        ("Tag 1", "2020-05-06", 6), ("Tag 1000", "2021-02-06", 6)],
    ["Tag", "Timestamp", "Value"])

df2 = spark.createDataFrame(
    [("Tag 1", "2020-05-02", "2020-05-03"), ("Tag 1000", "2021-02-03", "2021-02-04")],
    ["Tag", "Start", "End"])

Desired Dataframe:

print(df1)

Tag       Timestamp  Value
Tag 1     2020-05-01 1
Tag 1     2020-05-02 2
Tag 1     2020-05-03 3
Tag 1     2020-05-04 4       #Notice day 5 and 6 are not in the df
Tag 1000  2020-02-01 1
Tag 1000  2020-02-02 2
Tag 1000  2020-02-03 3
Tag 1000  2020-02-04 4
Tag 1000  2020-02-05 5       #Notice day 6 are not in the df

Doing this will only give me the dates that I needed based on the second dataframe and will eliminate 1,000,000's of rows that will not be analyzing.

So far what I understand is creating the window.

w = Window().partitionBy("Tag").orderBy("Timestamp")

Answer 1

You need to convert column timestamp , start , and end to DateType first using to_date function, add padding days using date_add , finally join both dataframes where date of timestamp column is between padded start and end

from pyspark.sql.functions import col, to_date, date_add

# convert to DateType
df1 = df1.withColumn('timestamp', to_date(col('Timestamp'), "yyyy-MM-dd"))
# convert to DateType then add padding days
df2 = (df2
       .withColumn("start_date", date_add(to_date(col('Start'), 'yyyy-MM-dd'), -2))
       .withColumn("end_date", date_add(to_date(col("End"), 'yyyy-MM-dd'), 1)))

df1 = df1.join(df2.withColumnRenamed('Tag', 'Tag2'),
               [col('Tag') == col('Tag2'), col('timestamp').between(col('start_date'), col('end_date'))],
               'left_semi')
df1.show()

+--------+----------+-----+
|     Tag| timestamp|Value|
+--------+----------+-----+
|Tag 1000|2021-02-01|    1|
|Tag 1000|2021-02-02|    2|
|Tag 1000|2021-02-03|    3|
|Tag 1000|2021-02-04|    4|
|Tag 1000|2021-02-05|    5|
|   Tag 1|2020-05-01|    1|
|   Tag 1|2020-05-02|    2|
|   Tag 1|2020-05-03|    3|
|   Tag 1|2020-05-04|    4|
+--------+----------+-----+