spark：纱线簇模式无法读取hdfs路径（无此类文件或目录）

Question

I'm using spark submit in yarn mode, but I'm getting this error:

Apparently my input path hdfs:// is missing one '/'
I'm passing hdfs://master:8020/usr/jimmy/Test/ , but the log output is hdfs:/master:8020/usr/jimmy/Test/Personal1 in the log file (the Personal1 is specified in my code)
The missing '/' is obviously the problem
What can I do?

Please help me

My command:

./bin/spark-submit --master yarn-cluster --class Movie.Movies /usr/jimmy/Move.jar hdfs://master:8020/usr/jimmy/Test/  hdfs://usr/jimmy/result 

Hadoop log:

15/09/08 16:33:54 ERROR yarn.ApplicationMaster: User class threw exception: hdfs:/master:8020/usr/jimmy/Test/Personal1 (No such file or directory)
java.io.FileNotFoundException: hdfs:/master:8020/usr/jimmy/Test/Personal1 (No such file or directory)

Update: I see this on spark official site.Does it means I have to use ntfs ???
My thought is : master and worker both can read file on hdfs path.And I try ( hdfs ls ) is OK. But I don't know why my execution has error.
Please guide me

Do I need Hadoop to run Spark?

No, but if you run on a cluster, you will need some form of shared file system (for example, NFS mounted at the same path on each node). If you have this type of filesystem, you can just deploy Spark in standalone mode.

Answer 1

The job is trying to access the input file Personal1 on hdfs.
Please check if you have configured the hdfs and pushed the file on it.

If yes, then try to locate Personal1 file through HDFS Web UI on this path /usr/jimmy/Test/
example: http://master:50070/explorer.html#/usr/jimmy/Test/

If you are able to locate the file then your job should locate the file as well.