Date Difference within Same Column Apache Spark

Question

I have one column and I need to find out date difference in days between each row, partitioned by Id.. This have to be done using Spark SQL. I have written below code but somehow the answer is coming wrong. Kindly let me know where am I going wrong.

WindowSpec window = Window.partitionBy("id").orderBy("date_time");
Dataset<Row> resultSet = testData.withColumn("day_diff", functions.datediff(col("date_time"), functions.lag(col("date_time"), 1).over(window)));

Answer 1

You should probably do it one by one.

• Step1 : Use window function to collect the Date from previous row.
• Step2 : Use datediff to find the difference .
• drop extra cols

testData
.withColumn("prev_date", functions.lag(col("date_time"),1).over(window))
.withColumn("day_diff", functions.datediff(col("date_time")), col("prev_date"))
.drop(col("prev_date"))