[英]Why saveAsTextFile do nothing?
I'm trying to implement simple WordCount in Scala + Spark. 我正在尝试在Scala + Spark中实现简单的WordCount。 Here is my code
这是我的代码
object FirstObject {
def main(args: Array[String]) {
val input = "/Data/input"
val conf = new SparkConf().setAppName("Simple Application")
.setMaster("spark://192.168.1.162:7077")
val sparkContext = new SparkContext(conf)
val text = sparkContext.textFile(input).cache()
val wordCounts = text.flatMap(line => line.split(" "))
.map(word => (word,1))
.reduceByKey((a,b) => a+b)
.sortByKey()
wordCounts.saveAsTextFile("/Data/output")
}
This job is working for 54s
, and finally do nothing. 这项工作持续了
54s
,最终什么也没做。 Is is not writing output to /Data/output
是不将输出写入
/Data/output
Also if I replace saveAsTextFile
with forEach(println)
it is produce desired output. 另外,如果我用
forEach(println)
替换saveAsTextFile
,则会产生所需的输出。
You should check your user rights for /data/output folder. 您应该检查/ data / output文件夹的用户权限。 This folder should have writing rights for your specific user.
该文件夹应具有针对您的特定用户的写权限。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.