Debug spark process on windows with Java 8

Question

I can run spark on windows with commands in cmd.exe:

set CLASS=org.apache.spark.deploy.SparkSubmit
%SPARK_HOME%\bin\spark-class2.cmd %CLASS%  --class "com.sel2in.TstFac" --master local[4]  target\scala-2.11\theApp_2.11-1.0.jar

In spark-class2.cmd want to add remote debugging options:

set JOPTS=
set JOPTS=-agentlib:jdwp=transport=dt_socket,server=y,address=8001,suspend=y,timeout=30
REM set JOPTS=-Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=y 
REM set JOPTS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,timeout=3,address=4002

But the settings don't work. It seems one program builds the actual command to build. Anyone been able to get the right set options in so can run the spark command in debug mode, attach to it like you would any Java process ? (local single spark mode)

Answer 1

Last two lines are helpful.

del %LAUNCHER_OUTPUT%
%SPARK_CMD%

So the command is written to a file. Then that file is read, joined and run!. Rem out the 2nd lsat line or add an echo and pause like

echo %LAUNCHER_OUTPUT%
pause
REM del %LAUNCHER_OUTPUT%
%SPARK_CMD%

OR

del %LAUNCHER_OUTPUT%
echo  %SPARK_CMD%
pause
%SPARK_CMD%

Then edit the read command to add debug options as you would to other java program. Then attach to it from eclipse.