在键值 RDD 中寻找最大值

Question

I have a key-value RDD of the form :

(Some(23661587),
CompactBuffer(Posting(2,23661643,Some(23661587),0,None), 
              Posting(2,23661682,Some(23661587),0,None)))

Here Some(23661587) is the key and data inside CompactBuffer is the value. I want to select the Posting type with maximum value for a particular attribute for each key.

How can I do that? I have limited experience in Scala and Spark. Thanks

Answer 1

I reproduced your example with some data.

As @sinanspd said, org.apache.spark.util.collection.CompactBuffer extends from scala.collection.immutable.Seq , you can follow this link CompactBuffer , so you can use methods from scala.collection.immutable.Seq Seq to sort the Seq and get the Posting max value.

My choice was Posting.value to sort the Seq but it could be value2 or any field in Posting class.

As an example

object FindingMaximum {

  val spark = SparkSession
    .builder()
    .appName("FindingMaximum")
    .master("local[*]")
    .getOrCreate()

  val sc = spark.sparkContext

  case class Posting(key: Int, value: Long, value2: Option[Long], value3: Int, value4: Option[Int])

  val data = List((Some(23661587),Seq(Posting(2,23661643,Some(23661587),0,None), Posting(2,23661682,Some(23661587),0,None))),
                  (Some(23661588),Seq(Posting(3,23661743,Some(23661588),0,None), Posting(3,23661682,Some(23661588),0,None))),
                  (Some(23661589),Seq(Posting(4,23661843,Some(23661589),0,None), Posting(4,23661882,Some(23661589),0,None))))

  def main(args: Array[String]): Unit = {

    sc.setLogLevel("ERROR")

    val rdd = sc.parallelize(data)

    val rddKeyMax = rdd.map({case(key, v) =>
      val max = v.sortBy(posting => posting.value).last
      (key, max)
    })
    rddKeyMax.foreach(println)
  }
}

/*
(Some(23661588),Posting(3,23661743,Some(23661588),0,None))
(Some(23661587),Posting(2,23661682,Some(23661587),0,None))
(Some(23661589),Posting(4,23661882,Some(23661589),0,None))
*/