将时间戳整数列转换为在火花数据帧中一小时后停止的日期时间

Question

I have a spark dataframe like that

I have timestamp as integer and I want to create another column with the entire date + the hour I don't understand why this code doesn't work :

df = df.withColumn("Date_per_hour", df.cast('date'))

Answer 1

You can cast the Date column to timestamp with the required format.

from pyspark.sql.functions import to_timestamp   

df = df.withColumn("new_date", to_timestamp("Date", "yyyyMMdd"))

Note* you asked for the hour also but I don't see that you have in your Date column hour information. If you have just add it to the format ,for example:

df = df.withColumn("new_date", to_timestamp("Date", "yyyyMMdd HHmmss"))