从 Spark DataFrame 列出的所有列的区别

Question

I have a data frame like below and I want to convert to expected format as distinct values in a list.

+---------------------+---------------+
|col1                 |col2           |
+---------------------+---------------+
|                  A  |             1 |
|                  B  |             2 |
|                  C  |             1 |
|                  D  |             1 |
|                  A  |             2 |
|               null  |             1 |
+---------------------+---------------+

Expected Format

+---------------------+---------------+
|col1                 |col2           |
+---------------------+---------------+
|      [A,B,C,D,null] |         [1,2] |
+---------------------+---------------+

Is there any wany to solve the above problem.??

Thanks in Advance !!

Answer 1

You can do something like this

import spark.implicits._

df
  .na.fill("null", Seq("col1"))
  .agg(
     func.collect_set($"col1").alias("col1"),
     func.collect_set($"col2").alias("col2")
  )