是rdd.contains函数在spark-scala中很昂贵

Question

I am getting millions of message from Kafka stream in spark-streaming. There are 15 different types of message. Messages come from a single topic. I can only differentiate message by its content. so I am using rdd.contains method to get the different type of rdd.

sample message

{"a":"foo", "b":"bar","type":"first" .......}
{"a":"foo1", "b":"bar1","type":"second" .......}
{"a":"foo2", "b":"bar2","type":"third" .......}
{"a":"foo", "b":"bar","type":"first" .......}
..............
...............
.........
so on

code

DStream.foreachRDD { rdd =>
  if (!rdd.isEmpty()) {
    val rdd_first = rdd.filter {
      ele => ele.contains("First")
    }
    if (!rdd_first.isEmpty()) {
      insertIntoTableFirst(hivecontext.read.json(rdd_first)) 
    }
    val rdd_second = rdd.filter {
      ele => ele.contains("Second")
    }
    if (!rdd_second.isEmpty()) {
     insertIntoTableSecond(hivecontext.read.json(rdd_second))
    }
         .............
         ......
    same way for 15 different rdd

is there any way to get different rdd from kafka topic message?

Answer 1

There's no rdd.contains . The function contains used here is applied to the String s in the RDD .

Like here:

val rdd_first = rdd.filter {
  element => element.contains("First") // each `element` is a String 
}

This method is not robust because other content in the String might meet the comparison, resulting in errors.

eg

{"a":"foo", "b":"bar","type":"second", "c": "first", .......}

One way to deal with this would be to first transform the JSON data into proper records, and then apply grouping or filtering logic on those records. For that, we first need a schema definition of the data. With the schema, we can parse the records into json and apply any processing on top of that:

case class Record(a:String, b:String, `type`:String)

import org.apache.spark.sql.types._
val schema = StructType(
               Array(
                StructField("a", StringType, true),
                StructField("b", StringType, true),
                StructField("type", String, true)
               )
             )

val processPerType: Map[String, Dataset[Record] => Unit ] = Map(...) 

stream.foreachRDD { rdd =>
  val records = rdd.toDF("value").select(from_json($"value", schema)).as[Record]
  processPerType.foreach{case (tpe, process) =>
      val target = records.filter(entry => entry.`type` == tpe)
      process(target)
  }
} 

The question does not specify what kind of logic needs to be applied to each type of record. What's presented here is a generic way of approaching the problem where any custom logic can be expressed as a function Dataset[Record] => Unit .

If the logic could be expressed as an aggregation, probably the Dataset aggregation functions will be more appropriate.