I am trying to utilise all the resources which I have on the cluster to run the Spark job. I have Cloudera Manager installed on all of the nodes. This is the command which I use to submit the job.
spark-submit --master yarn
--deploy-mode cluster
file:///[spark python file]
file://[app argument 1]
file://[app argument 2]
During the execution I receive following error:
diagnostics: Application application_1450777964379_0027 failed 2 times due to AM Container for appattempt_1450777964379_0027_000002 exited with exitCode: 1
Any ideas how to fix it will be much appreciated.
EDIT 1 The machine where Spark is installed is not accessible by WEB UI I tried to download the sources and read little bit more about the exception.
------------------------------------------------------------
| Job | Description |
------------------------------------------------------------
| 0 | saveAsTextFile at NativeMethodAccessorImpl.java:-2 |
------------------------------------------------------------
Taken from here ,
If the path starts with file:// or hdfs://, the path becomes ile:// or dfs://. If the path is absolute the first slash is removed.
There is no particular reason for it and needs to be fixed.
Try using an absolute path instead of file://
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.