[英]how to run sc.textFile() inside a foreach loop and do a union on it?
so this is what I have been trying and I'm a newbie here working with spark! 所以这就是我一直在努力的工作,我是火花的新手!
I'm trying to execute this code 我正在尝试执行此代码
val ii=sc.parallelize(Seq(("e.txt"),("r.txt"))).foreach{i => sc.textFile(i)}
but I'm getting "Nullpointer exception" 但我收到“ Nullpointer异常”
Thanks! 谢谢!
You can just add multiple files to the sc.textFile
. 您可以将多个文件添加到
sc.textFile
。 You should not use the sc
inside of a map operation. 您不应在地图操作中使用
sc
。 The map
function will be distributed to the different executors, and the sc
lives in the driver. map
函数将分配给不同的执行器,而sc
驻留在驱动程序中。 Therefore it will throw a Nullpointer exception. 因此,它将引发Nullpointer异常。
a.txt contents:
a.txt:line1
a.txt:line2
b.txt contents:
b.txt:line1
b.txt:line2
Spark allows you to add more files in the same operation: Spark允许您在同一操作中添加更多文件:
scala> sc.textFile("a.txt,b.txt").collect()
res1: Array[String] = Array(a.txt:line1, a.txt:line2, b.txt:line1, b.txt:line2)
Hope this helps and have fun with Spark! 希望这会有所帮助,并与Spark玩得开心!
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.