[英]Map Partition Iterator return
Can anyone help in accepting the returning Iterator listWords() method to mapPartitions. 任何人都可以帮助接受返回给MapPartitions的Iterator listWords()方法。
object MapPartitionExample {
def main(args: Array[String]): Unit = {
val conf= new SparkConf().setAppName("MapPartitionExample").setMaster("local[*]")
val sc= new SparkContext(conf)
val input:RDD[String] = sc.parallelize(List("ABC","DEF","GHU","YHG"))
val x= input.mapPartitions(word => listWords(word))
}
def listWords(words: Iterator[String]) : util.Iterator[String] = {
val arrList = new util.ArrayList[String]()
while( words.hasNext ) {
arrList.add( words.next())
}
return arrList.iterator()
}
}
Return type of the function used in mapPartitions
should be scala.collection.Iterator
, not java.util.Iterator
. mapPartitions
使用的函数的返回类型应该是scala.collection.Iterator
,而不是java.util.Iterator
。 I don't see much point of your current code, but you can use Scala mutable collections: 我看不到您当前代码的重点,但是您可以使用Scala可变集合:
import scala.collection.mutable.ArrayBuffer
def listWords(words: Iterator[String]) : Iterator[String] = {
val arr = ArrayBuffer[String]()
while( words.hasNext ) {
arr += words.next()
}
arr.toIterator
}
Personally I'd just map
: 我个人只是map
:
def listWords(words: Iterator[String]) : Iterator[String] = {
// Some init code
words.map(someFunction)
}
Iterable[NotInferU]
is expected but you are returning java.util.Iterator[String]
Iterable[NotInferU]
是Iterable[NotInferU]
,但是您要返回java.util.Iterator[String]
You would need to convert the java.util.Iterator
to scala Iterator
by importing scala.collection.JavaConversions._
as below 您需要通过如下所示导入scala.collection.JavaConversions._
将java.util.Iterator
转换为scala Iterator
。
def listWords(words: Iterator[String]) : Iterator[String] = {
val arrList = new util.ArrayList[String]()
while( words.hasNext ) {
arrList.add( words.next())
}
import scala.collection.JavaConversions._
return arrList.toList.iterator
}
Rest of the codes are as it is. 其余代码保持不变。
I hope the answer is helpful 我希望答案是有帮助的
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.