How can I verify that DSE Spark Shell is distributing across the cluster

Question

Is it possible to verify from within the Spark shell what nodes if the shell is connected to the cluster or is running just in local mode? I'm hoping to use that to investigate the following problem:

I've used DSE to setup a small 3 node Cassandra Analytics cluster. I can log onto any of the 3 servers and run dse spark and bring up the Spark shell. I have also verified that all 3 servers have the Spark master configured by running dsetool sparkmaster .

However, when I run any task using the Spark shell, it appears that the it is only running locally. I ran a small test command:

val rdd = sc.cassandraTable("test", "test_table")
rdd.count

When I check the Spark Master webpage, I see that only one server is running the job.

I suspect that when I run dse spark it's running the shell in local mode. I looked up how to specific a master for the Spark 0.9.1 shell and even when I use MASTER=<sparkmaster> dse spark (from the Programming Guide ) it still runs only in local mode.

Answer 1

Here's a walkthrough once you've started a DSE 4.5.1 cluster with 3 nodes, all set for Analytics Spark mode.

Once the cluster is up and running, you can determine which node is the Spark Master with command dsetool sparkmaster . This command just prints the current master; it does not affect which node is the master and does not start/stop it.

Point a web browser to the Spark Master web UI at the given IP address and port 7080. You should see 3 workers in the ALIVE state, and no Running Applications. (You may have some DEAD workers or Completed Applications if previous Spark jobs had happened on this cluster.)

Now on one node bring up the Spark shell with dse spark . If you check the Spark Master web UI, you should see one Running Application named "Spark shell". It will probably show 1 core allocated (the default).

If you click on the application ID link ("app-2014...") you'll see the details for that app, including one executor (worker). Any commands you give the Spark shell will run on this worker.

The default configuration is limiting the Spark master to only allowing each application to use 1 core, therefore the work will only be given to a single node.

To change this, login to the Spark master node and sudo edit the file /etc/dse/spark/spark-env.sh . Find the line that sets SPARK_MASTER_OPTS and remove the portion -Dspark.deploy.defaultCores=1 . Then restart DSE on this node ( sudo service dse restart ).

Once it comes up, check the Spark master web UI and repeat the test with the Spark shell. You should see that it's been allocated more cores, and any jobs it performs will happen on multiple nodes.

In a production environment you'd want to set the number of cores more carefully so that a single job doesn't take all the resources.