How to configure DSE Spark for remote debug

Question

I am using DataStax and trying to configure Spark for remote debug.

I have followed below steps : a) Went to dse-5.1/resources/spark/conf/dse-spark-env.sh b) Added below snippet so spark driver start in debug mode

export SPARK_SUBMIT_OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=4000 "

But spark does not start in debug more. Please suggest link/steps to be followed.

$ netstat -na | grep "7080"

tcp        0      0 0.0.0.0:7080            0.0.0.0:*               LISTEN

$netstat -na | grep "4000"

$

Answer 1

Spark application consists of driver program and number of executors, you should use different options for them: Debugging Spark Applications

To simplify things you can run application in local mode. There will be only one JVM for both driver and executors. Here is DSE example for this:

dse spark --master local[1] --driver-java-options -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=4000