Log4j filerotation doesn't work in spark-yarn

Question

I submit my application to yarn with a custom log4j property. The custom config itself is recognized and used by both the driver and executor except the log rotation part. (The log rotation itself works well without yarn)

Hadoop: 2.7.0
Spark : 3.1.1
OS: Windows10

spark-submit:

./bin/spark-submit.cmd --class scala_spark.ScalaApp --master yarn --files "./myapp/log4j-test.properties" --conf "spark.driver.extraJavaOptions=-Dlog4j.configuration=file:log4j-test.properties" --conf "spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j-test.properties" --deploy-mode cluster ./myapp/myApp.jar

log4j-test.properties:

# Set root logger level to DEBUG and its only appender to A1.
log4j.rootLogger=INFO, A1, my_file

log4j.logger.scala_spark=DEBUG

 # A1 Appender
log4j.appender.A1=org.apache.log4j.ConsoleAppender       
log4j.appender.A1.layout=org.apache.log4j.PatternLayout
log4j.appender.A1.layout.ConversionPattern=%-4r [%t] %-5p %c %x - %m%n

# file Appender
log4j.appender.my_file=org.apache.log4j.FileAppender
log4j.appender.my_file.append=true
log4j.appender.my_file.file=./myApp.log
log4j.appender.my_file.threshold=INFO
log4j.appender.my_file.layout=org.apache.log4j.PatternLayout
log4j.appender.my_file.layout.ConversionPattern=%-5p %c: %m%n

Result:
In $HADOOP_HOME\logs\userlogs\application_1617490627891_0001\container_1617490627891_0001_01_000001 only stderr and stdout can be found but not the myApp.log

NOTE:
I am 100% sure that log4j-test.properties is in effect because when I change something like rootlogger to TRACE, then those extra TRACE,DEBUG logs will appear in stdout. If I change to log4j.rootLogger=INFO, my_file, then nothing will printed to stdout ofcz, but myApp.log still nowhere.

EDIT:
I thought maybe the app can't create the file for some reason ( like permission issue), but there are no errors at all in the app,spark,yarn,hdfs logs.

Answer 1

Solution :
I had the change the log file location to:

log4j.appender.file_appender.file=${spark.yarn.app.container.log.dir}/myApp.log

If you need a reference to the proper location to put log files in the YARN so that YARN can properly display and aggregate them, use spark.yarn.app.container.log.dir in your log4j.properties

source: https://spark.apache.org/docs/latest/running-on-yarn.html