在 Scala 中将 IndexedSequence[Map[String, Map[String, Int]]] 转换为 Map[String, Map[String, Int]]

Question

I am a beginner in Scala and i have a case where i am trying to fetch a particular column in a dataframe and pass it to another method for further processing.

In this case, i am able to get the column values as String. I can do either of the two:

1. Convert String to Map[String, Map[String, Int]]

2. Convert IndexedSequence[Map[String, Map[String, Int]]] to Map[String, Map[String, Int]]

The value in my dataframe is as follows:

[endGoal -> [a -> 10, b -> 10], max -> [a -> 30, b -> 30]]

The code snippet am trying to use to convert the string to Map[String, Map[String, Int]] is as follows:

val map = recordArray(33) //Got the String
//Converting it to IndexedSequence[Map[String, Map[String, Int]]]
val result = for (line <- map; array = map.split(",").map(_.trim))
          yield Map(array.head -> array.tail.map(x => {val y = x.split(":"); (y(0).toString, y(1).toInt)}).toMap)

But the above code helps me get it converted only to IndexedSequence[Map[String, Map[String, Int]]]

Am stuck in getting it converted to Map[String, Map[String, Int]]. Could you please explain me with an example on how to achieve the value in the required type?

Thank you.

Answer 1

Basically, i was taking a wrong approach to solve the problem.. My bad.. I was trying to do something like this:

data.foreachPartition { rddpartition =>
    rddpartition.foreach { record =>
        var recordString = record.mkString(“,”)
      var recordArray = recordString.split(“,”)

Which is where i ended up getting a string and then i was stuck trying to convert the string to Map[String, Map[String, Int]] and ended up getting IndexedSequence[Map[String, Map[String, Int]]].

Found 2 solutions:

1. From what i have tried above, adding:

    result.toList.flatten.toMap

converts the IndexedSequence[Map[String, Map[String, Int]]] to Map[String, Map[String, Int]]

2. I should directly get the Map[String, Map[String, Int]] from the dataframe by doing this:

    data.foreachPartition { rddpartition => rddpartition.foreach { record => val uid = record.getAs[String]("uid") val advertiserId = record.getAs[String]("advertiserId") val time = record.getAs[Long]("time") val map = record.getAs[Map[String, Map[String, Int]]]("map") val crossDeviceUsers = record.getAs[mutable.WrappedArray[String]]("crossDeviceUsers")