Why can't I submit my DSE Spark application to my Spark cluster?

Question

Why can't I submit my DSE Spark application to the cluster? I can just run it at my local machine but not in the cluster.

This is the run command and everytime I run it it tells me could not connect to akka and I don't know why:

   dse spark-submit --master spark://localhost:7077 --executor-memory 10G --total-executor-cores 4 --driver-memory 1G  --packages org.apache.spark:spark-streaming-kafka_2.10:1.4.1 --jars /root/spark-streaming-kafka_2.10-1.4.1.jar  /root/pythonspark/com/spark/toutiaospark.py appname source

This is the error message :

Ivy Default Cache set to: /root/.ivy2/cache
The jars for the packages stored in: /root/.ivy2/jars
:: loading settings :: url = jar:file:/usr/share/dse/spark/lib/ivy-2.4.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
org.apache.spark#spark-streaming-kafka_2.10 added as a dependency
:: resolving dependencies :: org.apache.spark#spark-submit-parent;1.0
 confs: [default]
 found org.apache.spark#spark-streaming-kafka_2.10;1.4.1 in central
 found org.apache.kafka#kafka_2.10;0.8.2.1 in central
 found com.yammer.metrics#metrics-core;2.2.0 in central
 found org.slf4j#slf4j-api;1.7.10 in central
 found org.apache.kafka#kafka-clients;0.8.2.1 in central
 found net.jpountz.lz4#lz4;1.2.0 in central
 found org.xerial.snappy#snappy-java;1.1.1.7 in central
 found com.101tec#zkclient;0.3 in central
 found log4j#log4j;1.2.17 in central
 found org.spark-project.spark#unused;1.0.0 in central
:: resolution report :: resolve 469ms :: artifacts dl 14ms
 :: modules in use:
 com.101tec#zkclient;0.3 from central in [default]
 com.yammer.metrics#metrics-core;2.2.0 from central in [default]
 log4j#log4j;1.2.17 from central in [default]
 net.jpountz.lz4#lz4;1.2.0 from central in [default]
 org.apache.kafka#kafka-clients;0.8.2.1 from central in [default]
 org.apache.kafka#kafka_2.10;0.8.2.1 from central in [default]
 org.apache.spark#spark-streaming-kafka_2.10;1.4.1 from central in [default]
 org.slf4j#slf4j-api;1.7.10 from central in [default]
 org.spark-project.spark#unused;1.0.0 from central in [default]
 org.xerial.snappy#snappy-java;1.1.1.7 from central in [default]
 ---------------------------------------------------------------------
 |                  |            modules            ||   artifacts   |
 |       conf       | number| search|dwnlded|evicted|| number|dwnlded|
 ---------------------------------------------------------------------
 |      default     |   10  |   0   |   0   |   0   ||   10  |   0   |
 ---------------------------------------------------------------------
:: retrieving :: org.apache.spark#spark-submit-parent
 confs: [default]
 0 artifacts copied, 10 already retrieved (0kB/12ms)
WARN  2016-02-29 12:38:48 org.apache.spark.deploy.client.AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
WARN  2016-02-29 12:38:48 Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /localhost:7077
WARN  2016-02-29 12:39:08 org.apache.spark.deploy.client.AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
WARN  2016-02-29 12:39:08 Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /localhost:7077
WARN  2016-02-29 12:39:28 org.apache.spark.deploy.client.AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
WARN  2016-02-29 12:39:28 Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /localhost:7077
ERROR 2016-02-29 12:39:48 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.
WARN  2016-02-29 12:39:48 org.apache.spark.scheduler.cluster.SparkDeploySchedulerBackend: Application ID is not initialized yet.
WARN  2016-02-29 12:39:48 org.apache.spark.deploy.client.AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@localhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@localhost:7077
ERROR 2016-02-29 12:39:48 akka.actor.OneForOneStrategy: null
java.lang.NullPointerException: null
 at org.apache.spark.deploy.client.AppClient$ClientActor$$anonfun$receiveWithLogging$1.applyOrElse(AppClient.scala:160) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at scala.runtime.AbstractPartialFunction$mcVL$sp.apply$mcVL$sp(AbstractPartialFunction.scala:33) ~[scala-library-2.10.5.jar:na]
 at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:33) ~[scala-library-2.10.5.jar:na]
 at scala.runtime.AbstractPartialFunction$mcVL$sp.apply(AbstractPartialFunction.scala:25) ~[scala-library-2.10.5.jar:na]
 at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:59) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at org.apache.spark.util.ActorLogReceive$$anon$1.apply(ActorLogReceive.scala:42) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:118) ~[scala-library-2.10.5.jar:na]
 at org.apache.spark.util.ActorLogReceive$$anon$1.applyOrElse(ActorLogReceive.scala:42) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at akka.actor.Actor$class.aroundReceive(Actor.scala:465) ~[akka-actor_2.10-2.3.4-spark.jar:na]
 at org.apache.spark.deploy.client.AppClient$ClientActor.aroundReceive(AppClient.scala:61) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516) [akka-actor_2.10-2.3.4-spark.jar:na]
 at akka.actor.ActorCell.invoke(ActorCell.scala:487) [akka-actor_2.10-2.3.4-spark.jar:na]
 at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238) [akka-actor_2.10-2.3.4-spark.jar:na]
 at akka.dispatch.Mailbox.run(Mailbox.scala:220) [akka-actor_2.10-2.3.4-spark.jar:na]
 at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393) [akka-actor_2.10-2.3.4-spark.jar:na]
 at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [scala-library-2.10.5.jar:na]
 at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [scala-library-2.10.5.jar:na]
 at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [scala-library-2.10.5.jar:na]
 at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [scala-library-2.10.5.jar:na]
WARN  2016-02-29 12:39:48 Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /localhost:7077
ERROR 2016-02-29 12:39:48 org.apache.spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalStateException: Cannot call methods on a stopped SparkContext
 at org.apache.spark.SparkContext.org$apache$spark$SparkContext$$assertNotStopped(SparkContext.scala:103) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at org.apache.spark.SparkContext.getSchedulingMode(SparkContext.scala:1504) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at org.apache.spark.SparkContext.postEnvironmentUpdate(SparkContext.scala:2032) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:543) ~[spark-core_2.10-1.4.2.2.jar:1.4.2.2]
 at com.datastax.bdp.spark.DseSparkContext$.apply(DseSparkContext.scala:42) [dse-spark-4.8.4.jar:4.8.4]
 at com.datastax.bdp.spark.DseSparkContext.apply(DseSparkContext.scala..

Answer 1

The important message is

WARN  2016-02-29 12:39:48 Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@localhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: Connection refused: /localhost:7077
ERROR 2016-02-29 12:39:48 org.apache.spark.SparkContext: Error initializing SparkContext.

This is letting you know that your Spark Master is unreachable at localhost (possibly because of how the master has bound to this machine). By default the Spark Master is bound to the listen address of C*. The easiest solution is to just not specify --master in your launch script. DSE automatically will set the Spark Master for you.

   dse spark-submit --executor-memory 10G --total-executor-cores 4 --driver-memory 1G  --packages org.apache.spark:spark-streaming-kafka_2.10:1.4.1 /root/pythonspark/com/spark/toutiaospark.py appname source