按火花 dataframe 中每一行的 map 类型列中的值排序

Question

Input DataSet

ab c

1 2 B:2-C:3

1 1 C:1-D:2

2 2 F:1

Expected Output DataSet

ab c

1 2 C:3-B:2

1 1 D:2-C:1

2 2 F:1

So,

1. c column needs to be sorted in desc order of value after:
2. Each key value pair in c is separated by -
3. There also might be only 1 key value in pair in c column.

Can anyone help me on this? With basic knowledge of spark and scala, I am unable to solve this for now.

Answer 1

Below should do the job:

import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.functions.col

object VerySimpleApp extends App {

  val spark = SparkSession.builder.appName("test-app").master("local[1]").getOrCreate()
  case class Record(a: Int, b: Int, c: String)
  val rows = Seq(
    Record(1, 1, "C:1-D:2"),
    Record(1, 2, "B:2-C:3"),
    Record(2, 2, "F:1")
  )
  val df = spark.createDataFrame(rows)
  df.selectExpr("a", "b", "c", "split(c, '[:-]') cs").
    selectExpr("*", "element_at(cs, 2) col1", "element_at(cs, 4) col2").
    orderBy(col("col1").desc, col("col2").desc).
    show()

}

+---+---+-------+------------+----+----+
|  a|  b|      c|          cs|col1|col2|
+---+---+-------+------------+----+----+
|  1|  2|B:2-C:3|[B, 2, C, 3]|   2|   3|
|  1|  1|C:1-D:2|[C, 1, D, 2]|   1|   2|
|  2|  2|    F:1|      [F, 1]|   1|null|
+---+---+-------+------------+----+----+