Quartz SLF4J Log4J 在运行时外部修改到 SocketAppender 的附加程序

Question

We have an issue with a single customer where Quartz jobs stop running soon after start up. Diagnosing the issue has been very frustrating because Quartz was not properly logging to disk as it usually does for this customer. After several days of hard debugging the production system and using jstack after jobs stop running I was able to observe the main quartz process is IO bound attempting to write to a socket appender... which we have not configured. This is a production QA'd application stack deployed to over 1000 servers yet this issue happens only with one customer on one of their two servers running our software.

After jobs stop executing jstack shows the following for our main quartz thread:

"CIPDCEScheduler_QuartzSchedulerThread" #50 prio=5 os_prio=0 cpu=2609.38ms elapsed=335.10s tid=0x0000026f7c819800 nid=0x1304 runnable  [0x000000fb84ffe000]
   java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.SocketDispatcher.write0(java.base@13.0.1/Native Method)
   at sun.nio.ch.SocketDispatcher.write(java.base@13.0.1/SocketDispatcher.java:54)
   at sun.nio.ch.NioSocketImpl.tryWrite(java.base@13.0.1/NioSocketImpl.java:399)
   at sun.nio.ch.NioSocketImpl.implWrite(java.base@13.0.1/NioSocketImpl.java:414)
   at sun.nio.ch.NioSocketImpl.write(java.base@13.0.1/NioSocketImpl.java:441)
   at sun.nio.ch.NioSocketImpl$2.write(java.base@13.0.1/NioSocketImpl.java:825)
   at java.net.Socket$SocketOutputStream.write(java.base@13.0.1/Socket.java:1007)
   at java.io.ObjectOutputStream$BlockDataOutputStream.drain(java.base@13.0.1/ObjectOutputStream.java:1883)
// IO BOUND !!!                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
   at java.io.ObjectOutputStream$BlockDataOutputStream.setBlockDataMode(java.base@13.0.1/ObjectOutputStream.java:1792)
   at java.io.ObjectOutputStream.writeSerialData(java.base@13.0.1/ObjectOutputStream.java:1496)
   at java.io.ObjectOutputStream.writeOrdinaryObject(java.base@13.0.1/ObjectOutputStream.java:1433)
   at java.io.ObjectOutputStream.writeObject0(java.base@13.0.1/ObjectOutputStream.java:1179)
   at java.io.ObjectOutputStream.writeObject(java.base@13.0.1/ObjectOutputStream.java:349)
   at <b>org.apache.log4j.net.SocketAppender</b>.append(SocketAppender.java:268)
//       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
   at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) 
   - locked <0x00000000c0a0d398> (a org.apache.log4j.net.SocketAppender)
   at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
   at org.apache.log4j.Category.callAppenders(Category.java:206)
   - locked <0x00000000c0a09ef8> (a org.apache.log4j.spi.RootLogger)
   at org.apache.log4j.Category.forcedLog(Category.java:391)
   at org.apache.log4j.Category.log(Category.java:856)
   at org.slf4j.impl.Log4jLoggerAdapter.debug(Log4jLoggerAdapter.java:205)
   at org.quartz.impl.jdbcjobstore.DBSemaphore.obtainLock(DBSemaphore.java:115)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3781)
   at org.quartz.impl.jdbcjobstore.JobStoreSupport.triggerFired(JobStoreSupport.java:2865)
   at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:319)

The main takeaways from the above stack are highlighted but include "BlockDataOutputStream.drain" (IO Bound) and "org.apache.log4j.net.SocketAppender". We have only configured a rolling file appender (which is no longer writing). The presence of an IO bound SocketAppender is baffling as we have never configured such an appender and it is IO bound blocking our threads as a result. Our log4j properties are as follows:

log4j.debug=TRUE
log4j.rootLogger=INFO, R
log4j.appender.R=org.apache.log4j.RollingFileAppender
log4j.appender.R.File=C:/CIP/Server/tmp/scheduler.log
log4j.appender.R.MaxFileSize=5MB
log4j.appender.R.MaxBackupIndex=5
log4j.appender.R.layout=org.apache.log4j.PatternLayout
log4j.appender.R.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss.SSSS} %p %t %c - %m%n

The tomcat logs show that it is processing this log4j configuration on startup:

log4j: Parsing for [root] with value=[INFO, R].
log4j: Level token is [INFO].
log4j: Category root set to INFO
log4j: Parsing appender named "R".
log4j: Parsing layout options for "R".
log4j: Setting property [conversionPattern] to [%d{yyyy-MM-dd HH:mm:ss.SSSS} %p %t %c - %m%n].
log4j: End of parsing for "R".
log4j: Setting property [maxBackupIndex] to [5].
log4j: Setting property [file] to [C:/CIP/Server/tmp/scheduler.log].
log4j: Setting property [maxFileSize] to [5MB].
log4j: setFile called: C:/CIP/Server/tmp/scheduler.log, true
log4j: setFile ended
log4j: Parsed "R" options.
log4j: Finished configuring.

It is my opinion that some how at runtime the logging configuration is being changed by some external source. All of our components are embedded in our application stack, not installed globally, which do not use any global configurations on the server. I have checked that no classpath or Java environment variables are defined globally on the server. The above log4j configuration will create an empty log file on startup but never write to it (but only on this single deployment).

This customer does have SolarWinds installed on this server (and not the other server they have which works fine). I can see SolarWinds has its own embedded Java stack including SLF4J / Log4J installed in C:\Windows\SysWOW64/ContegoSPOP. However, I see no indication that it has any global footprint that should interfere with our software.

Does Tomcat and/or SLF4J have some mechanism for discovery or inheritance or modification from a remote process which could change the logging appenders for a Quartz servlent?

OpenJDK 13 + Tomcat 8.5.73 + Quartz 1.8.6 + SLF4J 1.6.0 + Log4J 1.2.17

Answer 1

This ended up being a very obscure issue likely to never impact others but I wanted to answer this just in case. We use PHPJavaBridge for some Java interop with PHP and within that WAR distribution is logic to try and auto-connect to Apache Chainsaw on port 4445. If it connects on that port it some how... magically... reroutes all logging to that port via a socket appender and discards your log4j configurations otherwise.

In our case an unrelated service was bound to port 4445 on the customer server and this odd feature kicked in and routed all of our logs to it. Since the service on that port was not reading data it would get IO bound and lock the jobs. We were able to disable the obscure feature by setting the following property in Tomcat at runtime:

-Dchainsaw.port=0

and I learned a lot along the way...