Run simple word count program using Apache Spark with python

Question

I have created a Apache Spark cluster (using pre-build for Hadoop distribution) having three machines, two are working as a compute node and one is working as master and compute node. I have installed all required software like Apache Hadoop, JAVA 8, Scala.

Now I want to run a simple word count program on my cluster. I copied program available at https://github.com/apache/spark/blob/master/examples/src/main/python/wordcount.py and saved it in a spark_wc.py.

I am running this script with argument as a file name which is already stored on Hadoop HDFS, it is giving error as

python spark_wc.py file.txt
2018-09-10 12:12:56 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Setting default log level to "WARN"
Traceback (most recent call last):
File "spark_wc.py", line 19, in <module>
lines = spark.read.text(sys.argv[1]).rdd.map(lambda r: r[0]) 
File "/usr/lib/python2.7/site-packages/pyspark/sql/readwriter.py", line 328, in text
return self._df(self._jreader.text(self._spark._sc._jvm.PythonUtils.toSeq(paths)))
 File "/usr/lib/python2.7/site-packages/py4j/java_gateway.py", line 1257, in __call__
     answer, self.gateway_client, self.target_id, self.name)
  File "/usr/lib/python2.7/site-packages/pyspark/sql/utils.py", line 69, in deco
    raise AnalysisException(s.split(': ', 1)[1], stackTrace)
 pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://master:9000/user/ncra/file.txt;'

Answer 1

from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql import Row

sc = SparkContext(conf=conf)
RddDataSet = sc.textFile("word_count.dat");
words = RddDataSet.flatMap(lambda x: x.split(" "))
result = words.map(lambda x: (x,1)).reduceByKey(lambda x,y: x+y)
result = result.collect()
for word in result:
    print("%s: %s" %(word[0], word[1]))

Answer 2

By seeing the error it seems the path is not present.

I suspect you have kept your file in local file system. If so try appending the path like below

file:/yourpath/filename.csv