如何将rdd行中的嵌套对象转换为某些自定义对象

Question

I'm trying to learn some scala/spark and trying to practice using some basic spark integration example. So my problem is that I have a Mongo db running locally. I'm pulling some data and making an rdd from it. The data in db has a structure like that:

{
    "_id": 0,
    "name": "aimee Zank",
    "scores": [
        {
            "score": 1.463179736705023,
            "type": "exam"
        },
        {
            "score": 11.78273309957772,
            "type": "quiz"
        },
        {
            "score": 35.8740349954354,
            "type": "homework"
        }
    ]
}

Here is some code:

val conf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("simple-app")
    val sparkSession = SparkSession.builder()
      .appName("example-spark-scala-read-and-write-from-mongo")
      .config(conf)
      .config("spark.mongodb.output.uri", "mongodb://sproot:12345@172.18.0.3:27017/spdb.students")
      .config("spark.mongodb.input.uri", "mongodb://sproot:12345@172.18.0.3:27017/spdb.students")
      .getOrCreate()

    // Reading Mongodb collection into a dataframe
    val df = MongoSpark.load(sparkSession)
    val dataRdd: RDD[Row] = df.rdd

    dataRdd.foreach(row => println(row.getValuesMap[Any](row.schema.fieldNames)))

The code above provides me this:

Map(_id -> 0, name -> aimee Zank, scores -> WrappedArray([1.463179736705023,exam], [11.78273309957772,quiz], [35.8740349954354,homework]))
Map(_id -> 1, name -> Aurelia Menendez, scores -> WrappedArray([60.06045071030959,exam], [52.79790691903873,quiz], [71.76133439165544,homework]))

At the end I have a problem converting this data to:

case class Student(id: Long, name: String, scores: Scores)

case class Scores(@JsonProperty("scores") scores: List[Score])

case class Score (
                 @JsonProperty("score") score: Double,
                 @JsonProperty("type") scoreType: String
)

To conclude - the problem is that I cannot convert some data from RDD to the Student object. The most problematic place for me is that 'scores' nested object. Please help me to understand how this should be done.

Answer 1

Played a bit more with it and ended up with the following solution:

object MainClass {

  def main(args: Array[String]): Unit = {

    val conf: SparkConf = new SparkConf().setMaster("local[*]").setAppName("simple-app")
    val sparkSession = SparkSession.builder()
      .appName("example-spark-scala-read-and-write-from-mongo")
      .config(conf)
      .config("spark.mongodb.output.uri", "mongodb://sproot:12345@172.18.0.3:27017/spdb.students")
      .config("spark.mongodb.input.uri", "mongodb://sproot:12345@172.18.0.3:27017/spdb.students")
      .getOrCreate()

    val objectMapper = new ObjectMapper()
    objectMapper.registerModule(DefaultScalaModule)

    // Reading Mongodb collection into a dataframe
    val df = MongoSpark.load(sparkSession)
    val dataRdd: RDD[Row] = df.rdd

    val students: List[Student] =
      dataRdd
        .collect()
        .map(row => Student(row.getInt(0), row.getString(1), createScoresObject(row))).toList
    println()
  }

  def createScoresObject(row: Row): Scores = {
    Scores(getAllScoresFromWrappedArray(row).map(x => Score(x.getDouble(0), x.getString(1))).toList)
  }

  def getAllScoresFromWrappedArray(row: Row): mutable.WrappedArray[GenericRowWithSchema] = {
    getScoresWrappedArray(row).map(x => x.asInstanceOf[GenericRowWithSchema])
  }

  def getScoresWrappedArray(row: Row): mutable.WrappedArray[AnyVal] = {
    row.getAs[mutable.WrappedArray[AnyVal]](2)
  }
}

case class Student(id: Long, name: String, scores: Scores)

case class Scores(scores: List[Score])

case class Score (score: Double, scoreType: String)

But I would be glad to know if there is some elegant solution.