SPARK N元语法和并行化未使用mapPartitions

Question

Problem at Hand Wrote an attempted improved bi-gram generator working over lines, taking into account full stops and the like. Results are as wanted. It does not use mapPartitions but is as per below.

import org.apache.spark.mllib.rdd.RDDFunctions._

val wordsRdd = sc.textFile("/FileStore/tables/natew5kh1478347610918/NGram_File.txt",10)  
val wordsRDDTextSplit = wordsRdd.map(line => (line.trim.split(" "))).flatMap(x => x).map(x => (x.toLowerCase())).map(x => x.replaceAll(",{1,}","")).map(x => x.replaceAll("!
{1,}",".")).map(x => x.replaceAll("\\?{1,}",".")).map(x => x.replaceAll("\\.{1,}",".")).map(x => x.replaceAll("\\W+",".")).filter(_ != ".")filter(_ != "")  

val x = wordsRDDTextSplit.collect() // need to do this due to lazy evaluation etc. I think, need collect()
val y = for ( Array(a,b,_*) <- x.sliding(2).toArray) 
yield (a, b) 
  val z = y.filter(x => !(x._1 contains ".")).map(x => (x._1.replaceAll("\\.{1,}",""), x._2.replaceAll("\\.{1,}","")))

I have some questions:

1. The results are as expected. No data is missed. But can I convert such an approach to a mapPartitions approach? Would I not lose some data? Many say that that this is the case due to the partitions that we would be processing having a subset of all the words and hence missing the relationship at a boundary of the split, ie.the next and the previous word. With a large file split I can see from the map point of view this could occur as well. Correct?

2. However, if you look at the code above (no mapPartitions attempt), it always works regardless of how much I parallelize this, 10 or 100 specified with partitions with words that are consecutive over different partitions. I checked this with mapPartitionsWithIndex. This I am not clear on. OK, a reduce on (x, y) => x + y is well understood.

Thanks in advance. I must be missing some elementary point in all this.

Output & Results z: Array[(String, String)] = Array((hello,how), (how,are), (are,you), (you,today), (i,am), (am,fine), (fine,but), (but,would), (would,like), (like,to), (to,talk), (talk,to), (to,you), (you,about), (about,the), (the,cat), (he,is), (is,not), (not,doing), (doing,so), (so,well), (what,should), (should,we), (we,do), (please,help), (help,me), (hi,there), (there,ged)) mapped: org.apache.spark.rdd.RDD[String] = MapPartitionsRDD[669] at mapPartitionsWithIndex at :123

Partition Assignment res13: Array[String] = Array(hello -> 0, how -> 0, are -> 0, you -> 0, today. -> 0, i -> 0, am -> 32, fine -> 32, but -> 32, would -> 32, like -> 32, to -> 32, talk -> 60, to -> 60, you -> 60, about -> 60, the -> 60, cat. -> 60, he -> 60, is -> 60, not -> 96, doing -> 96, so -> 96, well. -> 96, what -> 96, should -> 122, we -> 122, do. -> 122, please -> 122, help -> 122, me. -> 122, hi -> 155, there -> 155, ged. -> 155)

May be SPARK is just really smart, smarter than I thought initially. Or may be not? Saw some stuff on partition preservation, some of it contradictory imho.

map vs mapValues meaning former destroys partitioning and hence single partition processing?

Answer 1

You can use mapPartitions on in place of any of the maps used to create wordsRDDTextSplit , but I don't really see any reason to. mapPartitions is most useful when you have a high initialization cost that you don't want to pay for every record in the RDD.

Whether you use map or mapPartitions to create wordsRDDTextSplit , your sliding window doesn't operate on anything until you create the local data structure x .