spark dataframe 列组和 collect_to_list 不按照 orderBy

Question

I have a spark 2.2.0 dataframe dtfBase1 as below. BAQ is ID, AAA is date and AAG is numeric value in double.

And I would like to convert it into the following. The value of AAG should be indexed according to the order of AAA.

I used the following code

val dtfBase2=dtfBase1.orderBy($"BAQ",$"AAA").groupBy("BAQ").agg(collect_list("AAG") as "arrAAG")

But apparently in dtfBase2 the values of AAG seemed followed a random index instead of AAA's order in the original dataframe. How I index elements in arrAAG according to the order of AAA?

Answer 1

Assuming you're on Spark 2.4+, you can use array_sort and array_join

val dtfBase2 = dtfBase1.groupBy("BAQ")
  .agg(array_sort(collect_list(struct('aaa, 'aag))) as "arrAAG")
  .select('baq, array_join($"arrAAG.aag", ",")  as "arrAAG")

It creates a struct with the AAA and AAG, collects those in the aggregate and then sorts. We then concatenate using array_join , but just on the AAG element of the struct.

Since you're on Spark 2.2, this version should work

val dtfBase2 = dtfBase1.groupBy("BAQ")
  .agg(sort_array(collect_list(struct('aaa, 'aag))) as "arrAAG")
  .select('baq, concat_ws(",", $"arrAAG.aag") as "arrAAG")

Answer 2

I did this and it worked out. Somehow by caching dtfBase1 with desired orderBy, the order was remembered somewhere and got passed to next step. Feel free to suggest something doing it in one line.

val dtfBase1=....orderBy($"BAQ",$"AAA").cache()
val dtfBase2=dtfBase1.orderBy($"BAQ",$"AAA").groupBy("BAQ").agg(collect_list("AAG") as "arrAAG")