集群部署模式下的spark-submit将应用程序ID提供给控制台

Question

I am stuck in one problem which I need to resolve quickly. I have gone through many posts and tutorial about spark cluster deploy mode, but I am clueless about the approach as I am stuck for some days.

My use-case :- I have lots of spark jobs submitted using 'spark2-submit' command and I need to get the application id printed in the console once they are submitted. The spark jobs are submitted using cluster deploy mode. ( In normal client mode , its getting printed )

Points I need to consider while creating solution :- I am not supposed to change code ( as it would take long time, cause there are many applications running ), I can only provide log4j properties or some custom coding.

My approach:-

1) I have tried changing the log4j levels and various log4j parameters but the logging still goes to the centralized log directory.

Part from my log4j.properties:-

log4j.logger.org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend=ALL,console
log4j.appender.org.apache.spark.scheduler.cluster.YarnClusterSchedulerBackend.Target=System.out

log4j.logger.org.apache.spark.deploy.SparkSubmit=ALL
log4j.appender.org.apache.spark.deploy.SparkSubmit=console

log4j.logger.org.apache.spark.deploy.SparkSubmit=TRACE,console
log4j.additivity.org.apache.spark.deploy.SparkSubmit=false

log4j.logger.org.apache.spark.deploy.yarn.Client=ALL
log4j.appender.org.apache.spark.deploy.yarn.Client=console


log4j.logger.org.apache.spark.SparkContext=WARN
log4j.logger.org.apache.spark.scheduler.DAGScheduler=INFO,console

log4j.logger.org.apache.hadoop.ipc.Client=ALL

2) I have also tried to add custom listener and I am able to get the spark application id after the applications finishes , but not to console.

Code logic :-

public void onApplicationEnd(SparkListenerApplicationEnd arg0) 
    {
         for (Thread t : Thread.getAllStackTraces().keySet()) 
         {
            if (t.getName().equals("main"))
            {
                System.out.println("The current state : "+t.getState());

                Configuration config = new Configuration();

                ApplicationId appId = ConverterUtils.toApplicationId(getjobUId);

                // some logic to write to communicate with the main thread to print the app id to console.
            }
         }
    }

3) I have enabled the spark.eventLog to true and specified a directory in HDFS to write the event logs from spark-submit command .

If anyone could help me in finding an approach to the solution, it would be really helpful. Or if I am doing something very wrong, any insights would help me.

Thanks.

Answer 1

After being stuck at the same place for some days, I was finally able to get a solution to my problem.

After going through the Spark Code for the cluster deploy mode and some blogs, few things got clear. It might help someone else looking to achieve the same result.

In cluster deploy mode, the job is submitted via a Client thread from the machine from which the user is submitting. Actually I was passing the log4j configs to the driver and executors, but missed out on the part that the log 4j configs for the "Client" was missing.

So we need to use :-

SPARK_SUBMIT_OPTS="-Dlog4j.debug=true -Dlog4j.configuration= <location> /log4j.properties" spark-submit <rest of the parameters>

Answer 2

To clarify:

1. client mode means the Spark driver is running on the same machine you ran spark submit from
2. cluster mode means the Spark driver is running out on the cluster somewhere

You mentioned that it is getting logged when you run the app in client mode and you can see it in the console. Your output is also getting logged when you run in cluster mode you just can't see it because it is running on a different machine.

Some ideas:

• Aggregate the logs from the worker nodes into one place where you can parse them to get the app ID.
• Write the appIDs to some shared location like HDFS or a database. You might be able to use a Log4j appender if you want to keep log4j.