[英]Pyspark convert Year and week number to week_start Date & week_end Date
我正在嘗試使用 Pyspark dataframe 將年和周數轉換為周開始日期和周結束日期。
+---------+
|year_week|
+---------+
| 2019-W51|
| 2019-W52|
| 2020-W01|
| 2020-W02|
| 2020-W03|
| 2020-W04|
| 2020-W05|
| 2020-W06|
| 2020-W07|
+---------+
當我嘗試應用以下代碼時,我收到 object 列不可迭代的錯誤
df = df.withColumn('week_start_date', df.year_week.apply(lambda x: datetime.datetime.strptime(d + '-1', "%Y-W%W-%w")))
錯誤
TypeError: 'Column' object is not callable
預期結果是:
+---------+----------+----------+
|year_week|week_start| week_end|
+---------+----------+----------+
| 2019-W51|2019-12-16|2019-12-22|
| 2019-W52|2019-12-23|2019-12-29|
| 2020-W01|2019-12-30|2020-01-05|
| 2020-W02|2020-01-06|2020-01-12|
| 2020-W03|2020-01-13|2020-01-19|
| 2020-W04|2020-01-20|2020-01-26|
| 2020-W05|2020-01-27|2020-02-02|
| 2020-W06|2020-02-03|2020-02-09|
| 2020-W07|2020-02-10|2020-02-16|
+---------+----------+----------+
根據 Someshwar 的響應,只需對 python 從 scala 進行輕微更改。
df1 = df.withColumn("week_start", F.to_date(F.concat(F.col("year_week"), F.lit("-1")), "YYYY-'W'ww-u")).withColumn("week_end", F.next_day(F.col("week_start"), "SUN"))
嘗試這個-
// week starting from monday, concat "-1", for tuesday "-2" etc.
val p = df2.withColumn("week_start", to_date(concat($"year_week", lit("-1")), "YYYY-'W'ww-u"))
.withColumn("week_end", next_day($"week_start", "SUN"))
p.show(false)
p.printSchema()
/**
* +---------+----------+----------+
* |year_week|week_start|week_end |
* +---------+----------+----------+
* |2019-W51 |2019-12-16|2019-12-22|
* |2019-W52 |2019-12-23|2019-12-29|
* |2020-W01 |2019-12-30|2020-01-05|
* |2020-W02 |2020-01-06|2020-01-12|
* |2020-W03 |2020-01-13|2020-01-19|
* |2020-W04 |2020-01-20|2020-01-26|
* |2020-W05 |2020-01-27|2020-02-02|
* |2020-W06 |2020-02-03|2020-02-09|
* |2020-W07 |2020-02-10|2020-02-16|
* +---------+----------+----------+
*
* root
* |-- year_week: string (nullable = true)
* |-- week_start: date (nullable = true)
* |-- week_end: date (nullable = true)
*/
聲明:本站的技術帖子網頁,遵循CC BY-SA 4.0協議,如果您需要轉載,請注明本站網址或者原文地址。任何問題請咨詢:yoyou2525@163.com.