Spark/Scala 将 [map of array] 转换为 [map of map]

Question

I am looking to change the way data is stored in one of my dataframe's column. The columnn "content-value" has currently this type:

 |-- content-value: map (nullable = true)
 |    |-- key: integer
 |    |-- value: array (valueContainsNull = true)
 |    |    |-- element: string (containsNull = true)

And the data is currently stored like that:

{4 -> [5191, 57, -46, POS2], 5 -> [5413, 56, 48, POS2], 2 -> [5421, -59, 47, POS2], 1 -> [5237, -59, -47, POS2], 3 -> [5153, -10, 42, POS1]} 

I would like to change that to a map of map that would look like:

{4 -> {value -> 5191, x -> 57, y -> -46, pos -> POS2}, 5 -> {value -> 5413, x -> 56, y -> 48, pos -> POS2}, 2 -> {value -> 5421, x -> -59, y -> 47, pos -> POS2}, 1 -> {value -> 5237, x -> -59, y -> -47, pos -> POS2}, 3 -> {value -> 5153, x -> -10, y -> 42, pos -> POS1}} 

I've tried creating a new column with the keys ["value", "x", "y", "pos"] and using map_from_array without success.

Would love some help !

Answer 1

With dataset:

import spark.implicits._

case class Value(value: String, x: String, y: String, pos: String)

val ds = spark.createDataset[Map[Int, Array[String]]](Seq(Map(4 -> Array("5191", "57", "-46", "POS2"))))

val dsFinal = 
  ds.map(el => el.flatMap {
     case (key, value) => Map(key -> Value(value(0), value(1), value(2), value(3)))})

It gives:

+----------------------------+
|value                       |
+----------------------------+
|{4 -> {5191, 57, -46, POS2}}|
+----------------------------+