从Pyspark访问HDFS失败

Question

I have installed Hadoop 2.7.3 and pyspark 2.2.0 on Ubuntu 17.04.

Both Hadoop and Pyspark seem to work properly on their own. However, I did not manage to get files from HDFS in Pyspark. When I try to get a file from HDFS I get the following error:

https://imgur.com/j6Dy2u7

I read in another post that the environmental variable HADOOP_CONF_DIR needs to be set to access the HDFS. I also did that (see next screenshot), but then I get another error and Pyspark doesn't work any more.

https://imgur.com/AMpJ6TB

If I delete the Environmental Variable, everything works as before.

How can I fix the issue to open files from HDFS in Pyspark? I have spent a long time on that and would highly appreciate any help!

Answer 1

尽管这个答案有点晚，但是您应该使用hdfs:///test/PySpark.txt （注意3 / s）。