How is the Spark master elected in a Datastax Enterprise cluster?

Question

How is the Spark master elected in a Datastax Enterprise cluster? I have looked at the configurations in /etc/dse/dse-env.sh and /etc/dse/spark/spark-defaults.conf and /etc/dse/spark/spark-env.sh . But I cannot find it any of those locations?

On our cluster the Spark master keeps on moving from one node to another after the restart of the services?

Answer 1

In DSE 4.6, the Spark Master / Hadoop Job Tracker (always on the same node) are determined by a round robin algorithm and are stored in jobtracker table. This may change in future releases.

You can check and see what your current master is is by using:

dsetool listjt

This will output your current master and the reserve master if one has been configured.

To configure a reserve master use:

dsetool reservejt

Output might look something like this:

 $ dsetool listjt DC                             JobTracker   
 --                             --            
 Analytics-Analytics            192.168.101.122 
 ------ Reserve-JT = 192.168.101.121