Spark Scala 时间戳对比

Question

I have to write a code in spark scala in which I have a dataframe with below values.

PIN	REPORT_DATE	JOB_NO	ISSUED_AT_TIME	COMMITED_AT_TIME
CLAAB90	2020-12-17 00:00:00	TEST1	2020-12-17 09:12:41	2020-12-17 11:10:12
CLAAB90	2020-12-17 00:00:00	TEST2	2020-12-17 11:10:08	2020-12-17 13:10:05
CLAAB90	2020-12-17 00:00:00	TEST3	2020-12-17 13:10:15	2020-12-17 15:10:15
CLAAC07	2020-12-17 00:00:00	TEST4	2020-12-17 11:00:00	2020-12-17 12:10:00
CLAAC07	2020-12-17 00:00:00	TEST8	2020-12-17 12:10:05	2020-12-17 12:15:00
CLAAB91	2020-12-17 00:00:00	TEST5	2020-12-17 10:12:41	2020-12-17 11:10:12
CLAAB91	2020-12-17 00:00:00	TEST6	2020-12-17 11:10:08	2020-12-17 13:10:05
CLAAB91	2020-12-17 00:00:00	TEST7	2020-12-17 13:10:00	2020-12-17 15:10:15

here the column ISSUED_AT_TIME and COMMITED_AT_TIME are timestamp values. I need to compare the value between COMMITED_AT_TIME of row 1 and ISSUED_AT_TIME of row 2 and see which is greater.

I have tried to do this by declaring two variables and a list and then applying if condition

var V_AWI_TIME_GLOBAL=""
var V_COM_TIME_GLOBAL=""
var list2_1 = df2.select("ISSUED_AT_TIME").map(r => r.getString(0)).collect.toList

if((list2_1(j) >= V_AWI_TIME_GLOBAL) & (list2_1(j) > V_COM_TIME_GLOBAL))

where the value of each of these variables is as below.

list2_1(j)=2020-12-17 11:10:12  //The List2_1 is a list converted from above dataframe with ISSUED_AT_TIME 
V_AWI_TIME_GLOBAL=2020-12-17 09:12:41
V_COM_TIME_GLOBAL=2020-12-17 11:10:12

Apparently it is treating each of these columns as string. If I do not apply getString(0) in the list creation from dataframe then it is creating a list with the type Any.

Answer 1

As I go through your question, I still didn't get what was the exact expected output, Here is what I understand and might help

newDF.withColumn("ISSUED_AT_TIME_LAG", 
  lag($"COMMITED_AT_TIME", 1).over(Window.partitionBy($"PIN").orderBy($"JOB_NO")))
    .withColumn("RESULT",
      when($"ISSUED_AT_TIME_LAG" > $"COMMITED_AT_TIME", $"ISSUED_AT_TIME_LAG").otherwise($"COMMITED_AT_TIME")
    ).drop($"ISSUED_AT_TIME_LAG")
    .show(false)

Output:

+-------+-------------------+------+-------------------+-------------------+-------------------+
|PIN    |REPORT_DATE        |JOB_NO|ISSUED_AT_TIME     |COMMITED_AT_TIME   |RESULT             |
+-------+-------------------+------+-------------------+-------------------+-------------------+
|CLAAC07|2020-12-17 00:00:00|TEST4 |2020-12-17 11:00:00|2020-12-17 12:10:00|2020-12-17 12:10:00|
|CLAAC07|2020-12-17 00:00:00|TEST8 |2020-12-17 12:10:05|2020-12-17 12:15:00|2020-12-17 12:15:00|
|CLAAB91|2020-12-17 00:00:00|TEST5 |2020-12-17 10:12:41|2020-12-17 11:10:12|2020-12-17 11:10:12|
|CLAAB91|2020-12-17 00:00:00|TEST6 |2020-12-17 11:10:08|2020-12-17 13:10:05|2020-12-17 13:10:05|
|CLAAB91|2020-12-17 00:00:00|TEST7 |2020-12-17 13:10:00|2020-12-17 15:10:15|2020-12-17 15:10:15|
|CLAAB90|2020-12-17 00:00:00|TEST1 |2020-12-17 09:12:41|2020-12-17 11:10:12|2020-12-17 11:10:12|
|CLAAB90|2020-12-17 00:00:00|TEST2 |2020-12-17 11:10:08|2020-12-17 13:10:05|2020-12-17 13:10:05|
|CLAAB90|2020-12-17 00:00:00|TEST3 |2020-12-17 13:10:15|2020-12-17 15:10:15|2020-12-17 15:10:15|
+-------+-------------------+------+-------------------+-------------------+-------------------+

Answer 2

I tried to compare the timestamp column by casting it to long datatype and then taking the difference. Mostly not the most efficient method but it worked in my case.

    //Create a view from the dataframe 
    df2.createOrReplaceGlobalTempView("RnView_row_1")
    //Create a dataframe followed by a view with 1st Row
    sql="select PIN as pin ,REPORT_DATE as report_date,JOB_NO as JOB_NO_FIRST,JOB_NO as 
    JOB_NO_LAST,ISSUED_AT_TIME as FINAL_AWI,COMMITED_AT_TIME as FINAL_COM,1 as 
    OVERLAP_ROWCOUNT_DIVISOR from global_temp.RnView_row_1 where row_number=%s".format(j)
    var temp_df_11=spark.sql(sql)
    temp_df_11.createOrReplaceGlobalTempView("row_11")
    //Create a dataframe followed by a view with 2nd Row
    sql="select PIN as pin ,REPORT_DATE as report_date,JOB_NO as JOB_NO_FIRST,JOB_NO as 
    JOB_NO_LAST,ISSUED_AT_TIME as FINAL_AWI,COMMITED_AT_TIME as FINAL_COM,1 as 
    OVERLAP_ROWCOUNT_DIVISOR from global_temp.RnView_row_1 where 
    row_number=%s".format(j+1)
    var temp_df_12=spark.sql(sql)
    temp_df_12.createOrReplaceGlobalTempView("row_12")
    //Create a dataframe by joining the two views to achive a single row with required 
    //values from row 1 and row 2
    sql="select row_11.pin, row_11.report_date,row_11.JOB_NO as JOB_NO_FIRST, 
    row_12.JOB_NO as JOB_NO_LAST ,row_12.FINAL_AWI as ROW_2_FINAL_AWI,row_12.FINAL_COM as 
    ROW_2_FINAL_COM, row_11.FINAL_COM as ROW_1_FINAL_COM,row_11.FINAL_AWI as 
    ROW_1_FINAL_AWI,1 as OVERLAP_ROWCOUNT_DIVISOR from global_temp.row_11 join  
    global_temp.row_12 on row_11.pin=row_12.pin"
    var temp_df_13=spark.sql(sql)
    temp_df_13.createOrReplaceGlobalTempView("Final_view_1")

    //Creating a dataframe where we cast the timestamp to long and take the difference
    var temp_df_1=temp_df_13.withColumn("ROW_2_FINAL_AWI",to_timestamp(col("ROW_2_FINAL_AWI"))).withColumn("ROW_1_FINAL_COM", 
    to_timestamp(col("ROW_1_FINAL_COM"))).withColumn("ROW_1_FINAL_AWI", 
    to_timestamp(col("ROW_1_FINAL_AWI"))).withColumn("ROW_2_FINAL_COM", 
to_timestamp(col("ROW_2_FINAL_COM"))).withColumn("Diff_In_row2AWI_and_row1_COM",col("ROW_2_FINAL_AWI").cast(LongType) - col("ROW_1_FINAL_COM").cast(LongType)).withColumn("Diff_In_row2AWI_and_row1_AWI",col("ROW_2_FINAL_AWI").cast(LongType) - col("ROW_1_FINAL_AWI").cast(LongType)).withColumn("Diff_In_row1COM_and_row2_COM",col("ROW_1_FINAL_COM").cast(LongType) - col("ROW_2_FINAL_COM").cast(LongType))

   //then compare if the difference is  greater than 0  or not to see which is greater
    if(Diff_In_row1COM_and_row2_COM_1>=0)
    {
    println("hello"
    }

The Output with difference after converting to Long is

pin	report_date	JOB_NO_FIRST	JOB_NO_LAST	ROW_2_FINAL_AWI	ROW_2_FINAL_COM	ROW_1_FINAL_COM	ROW_1_FINAL_AWI	OVERLAP_ROWCOUNT_DIVISOR	Diff_In_row2AWI_and_row1_COM	Diff_In_row2AWI_and_row1_AWI	Diff_In_row1COM_and_row2_COM
CLAAB90	2020-12-17 00:00:00	TEST1	TEST2	2020-12-17 11:10:08	2020-12-17 13:10:05	2020-12-17 11:10:12	2020-12-17 09:12:41	1	-4	7047	-7193