Azure HD Insight - YARN UI is not showing logs on stderr suddenly

Question

We used to see our application logs by going to logs-->stderr and it used to show SPARK INFO logs along with our own INFO logs, for last 1 month it is not showing anything but this:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/spark2/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/hdp/4.1.0.26/hadoop/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

We are using Azure HDI for last 2+ years, since last 3-4 weeks YARN UI is not printing logs on stderr.We tried all options like using old HDI version, using our old jar file, research on log4j properties, nothing worked. We are guessing that somehow log4j output is getting redirected to somewhere else. But we never changed any config properties and dont know what to change. We also tried submitting the job using Livy as well as spark-submit, same issue.

But we should mention that if we use "println()" it works and prints on stderr.

Appreciate any thread/help please.

Answer 1

You can try a few things:

• Check whether the Spark History server is running fine in Ambari
• In Ambari => Spark2 Find below config keys and add a console to anywhere that has the - Dlog4jspark.root.logger=INFO,RFA,Anonymizer to Dlog4jspark.root.logger=INFO,console,RFA,Anonymizer

You can find it in the below properties spark.executor.extraJavaOptions , content , spark.driver.extraJavaOptions

• Whether you are able to retrieve anything using

   yarn logs -applicationId <appid> --appOwner <userid>

Or

   yarn logs -applicationId <appid> 

In case if it is not getting displayed, then you need to check the permission of the user in the directory and also space left and used ( du -h ) for log4j properties ( spark.yarn.app.container.log.dir ). You can refer here

• You need to check if you are still facing the issue with other users.
• Try with a small Spark job with deploy mode as a client since its cluster mode expects the log4j to be present in every node.
• As a workaround, you can use Custom log4j.Link here