Enable remote JMX monitoring for Spark running on DataStax Enterprise

Question

What steps are required to perform on an standard AMI DSE installation on AWS EC2 to enable remote JMX monitoring?

I did following steps but I am not able to connect using VisualVM:

1. Create /etc/dse/spark/conf/metrics.properties
2. Copy&paste contents of https://github.com/apache/spark/blob/branch-1.4/conf/metrics.properties.template
3. Uncommented these lines:
   • org.apache.spark.metrics.sink.JmxSink
   • *.sink.jmx.class=org.apache.spark.metrics.sink.JmxSink
   • master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
   • worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
   • driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
   • executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
4. Restarted DSE cluster via OpsCenter

Answer 1

By default, remote jmx monitoring is turned off in DSE 4.8.0 and up. Assuming that the ami is running a recent version of DSE. Then spark will be running on a DSE node. The docs are here but here's a summary:

• edit cassandra-env.sh and set the following:

   JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.authenticate=true" JVM_OPTS="$JVM_OPTS -Dcom.sun.management.jmxremote.password.file=/etc/cassandra/jmxremote.password" LOCAL_JMX=no` 

• copy jmxremote.password.template from your jre/jdk install location to /etc/cassandra and rename it to jmxremote.password

   chown cassandra:cassandra /etc/cassandra/jmxremote.password chmod 400 /etc/cassandra/jmxremote.password` 

• Edit jmxremote.password and user and password.

   monitorRole QED controlRole R&D cassandra cass_password` 

• Sorry about this one, but you have to add the user with r/w permissions to:

   /usr/lib/jvm/java-8-oracle/jre/lib/management/jmxremote.access 

  I did this install on Ubuntu with Oracle Java 8 installed. Please change the path to match your java installation.

   monitorRole readonly cassandra readwrite controlRole readwrite \\ create javax.management.monitor.,javax.management.timer. \\ unregister 

Restart cassandra and attach your jmx tools using the cassandra username and password.