在YARN上运行Spark作业

Question

I am trying to utilise all the resources which I have on the cluster to run the Spark job. I have Cloudera Manager installed on all of the nodes. This is the command which I use to submit the job.

spark-submit --master yarn 
             --deploy-mode cluster 
             file:///[spark python file]
             file://[app argument 1]
             file://[app argument 2]

During the execution I receive following error:

diagnostics: Application application_1450777964379_0027 failed 2 times due to AM Container for appattempt_1450777964379_0027_000002 exited with  exitCode: 1

Any ideas how to fix it will be much appreciated.

EDIT 1 The machine where Spark is installed is not accessible by WEB UI I tried to download the sources and read little bit more about the exception.

------------------------------------------------------------
| Job | Description                                        | 
------------------------------------------------------------
| 0   | saveAsTextFile at NativeMethodAccessorImpl.java:-2 | 
------------------------------------------------------------

Answer 1

Taken from here ,

If the path starts with file:// or hdfs://, the path becomes ile:// or dfs://. If the path is absolute the first slash is removed.

There is no particular reason for it and needs to be fixed.

Try using an absolute path instead of file://