Running Applications doesn t appear spark web Ui but runs

Question

i need your help, i created 2 apps (one which using spray framework and the other one receive messages from kafka and send it to cassandra). Both run all the time and should never stop. I m in standalone on the server and my conf is :

- In spark_env.sh :

SPARK_MASTER_IP=MYIP
SPARK_EXECUTOR_CORES=2
SPARK_MASTER_PORT=7077
SPARK_EXECUTOR_MEMORY=4g
#SPARK_WORKER_PORT=65000
MASTER=spark://${SPARK_MASTER_IP}:${SPARK_MASTER_PORT}
SPARK_LOCAL_IP=MYIP
SPARK_MASTER_WEBUI_PORT=8080

- In spark_env.sh :
spark.master                     spark://MYIPMASTER:7077
spark.eventLog.enabled           true
spark.eventLog.dir               /opt/spark-1.6.1-bin-hadoop2.6/spark-events
spark.history.fs.logDirectory    /opt/spark-1.6.1-bin-hadoop2.6/logs
spark.io.compression.codec       lzf
spark.cassandra.connection.host MYIPMASTER
spark.cassandra.auth.username   LOGIN
spark.cassandra.auth.password   PASSWORD

I can access on both pages : MYIP:8080/ and MYIP:4040/ But on http://MYIP:8080/ , i see only my workers , i can t see my application which running.

When i submit i use this :

/opt/spark-1.6.1-bin-hadoop2.6/bin/spark-submit --class MYCLASS --verbose --conf spark.eventLog.enable=true --conf spark.master.ui.port=8080 --master local[2] /opt/spark-1.6.1-bin-hadoop2.6/jars/MYJAR.jar

Why ? Could you help me?

Thanks a lot :)

Answer 1

In your spark-submit command you are using the --master as local[2] which is submitting the application in local mode. If you wants to run it on the standalone cluster that you are running then you should pass spark master URL in master option ie --master spark://MYIPMASTER:7077

Answer 2

In terms of the master, spark-submit will respect the setting by following orders,

1. The master URL in your application code, which is the SparkSession.builder().master("...")
2. The --master parameter for the spark-submit command
3. The default configuration in your spark-defaults.conf

Answer 3

Mode: Standalone cluster

1> bin/spark-submit --class com.deepak.spark.App ../spark-0.0.2-SNAPSHOT.jar --master spark://172.29.44.63:7077 , was not working because master was specified after the jar

2> bin/spark-submit --class com.deepak.spark.App --master spark://172.29.44.63:7077 ../spark-0.0.2-SNAPSHOT.jar , this worked