在 Spark/Scala 中使用 date_format 将时间戳转换为星期几字符串

Question

I keep on getting error, that I pass too many arguments and not sure why, as I am following the exact examples from:

command-629173529675356:9: error: too many arguments for method apply: (colName: String)org.apache.spark.sql.Column in class Dataset
val df_2 = date_format.withColumn("day_of_week", date_format(col("date"), "EEEE"))

My code:

val date_format = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = date_format.withColumn("day_of_week", date_format(col("date"), "EEEE"))

Thank you for the help!

Answer 1

dayofweek是你正在寻找的功能，所以像这样

date_format.withColumn("day_of_week", dayofweek(col("date")))

Answer 2

You get your error because you named your first dataframe date_format , which is the same name as the Spark's built-in function you want to use. So when you call date_format , you're retrieving your dataframe instead of date_format built-in function.

To solve this, you should either rename your first dataframe:

val df_1 = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = df_1.withColumn("day_of_week", date_format(col("date"), "EEEE"))

Or ensure that you're calling right date_format by importing functions and then call functions.date_format when extracting day of week:

import org.apache.spark.sql.functions

val date_format = df_filter.withColumn("date", to_date(col("pickup_datetime")))
val df_2 = date_format.withColumn("day_of_week", functions.date_format(col("date"), "EEEE"))