如何在集群模式下将Spark应用提交到YARN？

Question

I have created a Spark WordCount application which I ran using spark-submit command in the shell in local mode.

When I try to run it in cluster mode on YARN using the command:

spark-submit --class com.WordCount --master yarn --deploy-mode cluster WordCount-1.0.jar

It does not seem to be running and shows status as:

Application report for application_1480577073003_0019 (state: ACCEPTED)

How to spark-submit the Spark application to YARN in cluster mode?

Answer 1

The reason for this issue is your application/driver is requesting more resources than the available resources in the cluster at that time.

Since you haven't specified any resource parameters, your driver will request resources with default values. That means your cluster is not able to provide the resources.

Possible reasons:

1. Your cluster doesn't have executors with enough memory/cores (default 1GB,1core)
2. Your cluster has executors with enough memory/cores but they are assigned to some other jobs.

Solution:

1. Either reduce default values of executor memory/cores request or increase memory/cores per yarn-container
2. Increase the cluster resources by adding more executors or wait for other jobs to complete[Or kill them if you don't like those jobs ;)]

Answer 2

After you spark-submit --deploy-mode cluster your Spark application, the driver and the executors are on the cluster's nodes.

From Spark's official documentation :

Deploy mode Distinguishes where the driver process runs. In "cluster" mode, the framework launches the driver inside of the cluster. In "client" mode, the submitter launches the driver outside of the cluster.

You'll get the application id being the handle to your application.

You should use yarn application -status command to check the status of a Spark application.

-status Prints the status of the application.