Java Spark 删除重复项/空值并保留顺序

Question

I have the below Java Spark dataset/dataframe.

Col_1 Col_2 Col_3 ...
A     1     1
A     1     NULL
B     2     2
B     2     3
C     1     NULL

There are close to 25 columns in this dataset and I have to remove those records which are duplicated on Col_1. If the second record is NULL, then NULL has to be removed (like in case of COl_1 = A) and if there are multiple valid values like in case of Col_1 = B then only one valid Col_2 = 2 and Col_3 = 2 should only be retained everytime. If there is only one record with null like in case of Col_1 = C. then it has to be retained

Expected Output:

Col_1 Col_2 Col_3 ...
A     1     1
B     2     2
C     1     NULL

What i tried so far:

I tried using group by and collect set with sort_array and array_remove but it removes the nulls altogether even if there is one row.

How to achieve the expected output in Java Spark.

Answer 1

This is how you can do it using spark dataframes:

import org.apache.spark.sql.functions.{coalesce, col, lit, min, struct}

val rows = Seq(
  ("A",1,Some(1)),
  ("A",1, Option.empty[Int]),
  ("B",2,Some(2)),
  ("B",2,Some(3)),
  ("C",1,Option.empty[Int]))
  .toDF("Col_1", "Col_2", "Col_3")

rows.show()

+-----+-----+-----+
|Col_1|Col_2|Col_3|
+-----+-----+-----+
|    A|    1|    1|
|    A|    1| null|
|    B|    2|    2|
|    B|    2|    3|
|    C|    1| null|
+-----+-----+-----+

val deduped = rows.groupBy(col("Col_1"))
  .agg(
    min(
      struct(
        coalesce(col("Col_3"), lit(Int.MaxValue)).as("null_maxed"), 
        col("Col_2"), 
        col("Col_3"))).as("argmax"))
  .select(col("Col_1"), col("argmax.Col_2"), col("argmax.Col_3"))

deduped.show()

+-----+-----+-----+
|Col_1|Col_2|Col_3|
+-----+-----+-----+
|    B|    2|    2|
|    C|    1| null| 
|    A|    1|    1|
+-----+-----+-----+

Whats happening here is you are grouping by Col_1 and then getting the minimum of a composite struct of Col_3 and Col_2 but nulls in Col_3 have been replaced with the max integer value so they don't impact the ordering. We then select the original Col_3 and Col_2 from the resulting row. I realise this is in scala but the syntax for java should be very similar.