如何在過濾條件 pyspark 中使用 function

Question

df=basedf.select("date","year","week","day").distinct()\
         .orderBy("date")\
         .withColumn("ISOWeek", concat("year","week"))\
         .filter("date"<=min_date())

def min_date():
    
    min_df=basedf.select("date")\
                 .orderBy("date")\
                 .agg(min("date").alias("date_min"))\
                 .select(date_add(col("date_min"),SelectedWeeks*7).alias("end_date"))
    return min_df

#selectedweeks=10 (No. of weeks selected from user)

代碼的兩個部分都單獨工作，但我想將結束日期傳遞給過濾條件。

Answer 1

SelectedWeeks=10 (No. of weeks selected from user)

def min_date():    
    min_df=basedf.select("date")\
                 .orderBy("date")\
                 .agg(min("date").alias("date_min"))\
                 .select(date_add(col("date_min"), SelectedWeeks*7))
    return min_df.collect()[0][0]

end_date = min_date()

df=basedf.select("date","year","week","day").distinct()\
         .orderBy("date")\
         .withColumn("ISOWeek", concat("year","week"))\
         .filter(col("date") < end_date)