在Spark中将地图转换为mapPartitions

Question

I have a code to analyze the log file using map transformatio n. Then the RDD got converted to DF .

val logData = sc.textFile("hdfs://quickstart.cloudera:8020/user/cloudera/syslog.txt")

val logDataDF = logData.map(rec => (rec.split(" ")(0), rec.split(" ")(2), rec.split(" ")(5))).toDF("month", "date", "process")

I would like to know whether I can use mapPartitions in this case instead of map .

Answer 1

I don't know what is your use case but you can definitely use mapPartition instead of map . Below code will return the same logDataDF .

val logDataDF = logData.mapPartitions(x => {
  val lst = scala.collection.mutable.ListBuffer[(String, String, String)]()
  while (x.hasNext) {
    val rec = x.next().split(" ")
    lst += ((rec(0), rec(2), rec(5)))
  }
  lst.iterator
}).toDF("month", "date", "process")