線程轉儲顯示線程未釋放鎖。

Question

• 如下圖在線程轉儲中，id為qtp336276309-556300線程獲得了org.apache.log4j.spi.RootLogger's鎖並且沒有釋放它。
  這會導致其他線程被阻塞。
• 這有時伴隨着主機上的 CPU 利用率在一周內從 5% 持續增加到 30%。
• 想知道處於阻塞狀態的線程是否會導致 CPU 峰值？ 如果是，那么我該如何解決問題？
• 如果不是，那么在解決 CPU 使用率飆升的情況下還應該檢查什么？

• 線程轉儲如下：

   "qtp336276309-561036" #561036 prio=5 os_prio=0 tid=0x00007efe80576800 nid=0x20a7 waiting for monitor entry [0x00007efe19fbd000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.log4j.Category.callAppenders(Category.java:204) - waiting to lock <0x00000000e02333c0> (a org.apache.log4j.spi.RootLogger) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:601) at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224) at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301) at java.util.logging.Logger.log(Logger.java:738) at java.util.logging.Logger.doLog(Logger.java:765) at java.util.logging.Logger.log(Logger.java:788) at java.util.logging.Logger.info(Logger.java:1490) at org.apache.mesos.chronos.scheduler.api.TaskManagementResource.updateStatus(TaskManagementResource.scala:43) "qtp336276309-561035" #561035 prio=5 os_prio=0 tid=0x00007efe80193000 nid=0x20a6 waiting for monitor entry [0x00007efe248f4000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.log4j.Category.callAppenders(Category.java:204) - waiting to lock <0x00000000e02333c0> (a org.apache.log4j.spi.RootLogger) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.info(Category.java:666) at mesosphere.chaos.http.ChaosRequestLog.write(ChaosRequestLog.scala:15) at org.eclipse.jetty.server.NCSARequestLog.log(NCSARequestLog.java:591) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None "Thread-328668" #561032 prio=5 os_prio=0 tid=0x00007efe50052000 nid=0x62 waiting for monitor entry [0x00007efe6b0cb000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.log4j.Category.callAppenders(Category.java:204) - waiting to lock <0x00000000e02333c0> (a org.apache.log4j.spi.RootLogger) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.log(Category.java:856) at org.slf4j.impl.Log4jLoggerAdapter.log(Log4jLoggerAdapter.java:601) at org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:224) at org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:301) at java.util.logging.Logger.log(Logger.java:738) at java.util.logging.Logger.doLog(Logger.java:765) at java.util.logging.Logger.log(Logger.java:788) at java.util.logging.Logger.info(Logger.java:1490) at org.apache.mesos.chronos.scheduler.mesos.MesosJobFramework.statusUpdate(MesosJobFramework.scala:224) at sun.reflect.GeneratedMethodAccessor89.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.google.inject.internal.DelegatingInvocationHandler.invoke(DelegatingInvocationHandler.java:37) at com.sun.proxy.$Proxy30.statusUpdate(Unknown Source) Locked ownable synchronizers: - None "qtp336276309-556300" #556300 prio=5 os_prio=0 tid=0x00007efe81654800 nid=0x2047 runnable [0x00007efe1a1c0000] java.lang.Thread.State: RUNNABLE at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) - locked <0x00000000e0234470> (a java.io.BufferedOutputStream) at java.io.PrintStream.write(PrintStream.java:480) - locked <0x00000000e0234450> (a java.io.PrintStream) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291) at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:295) at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:141) - locked <0x00000000e0234438> (a java.io.OutputStreamWriter) at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:229) at org.apache.log4j.helpers.QuietWriter.flush(QuietWriter.java:59) at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:324) at org.apache.log4j.WriterAppender.append(WriterAppender.java:162) at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251) - locked <0x00000000e0233cc0> (a org.apache.log4j.ConsoleAppender) at org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66) at org.apache.log4j.Category.callAppenders(Category.java:206) - locked <0x00000000e02333c0> (a org.apache.log4j.spi.RootLogger) at org.apache.log4j.Category.forcedLog(Category.java:391) at org.apache.log4j.Category.info(Category.java:666) at mesosphere.chaos.http.ChaosRequestLog.write(ChaosRequestLog.scala:15) at org.eclipse.jetty.server.NCSARequestLog.log(NCSARequestLog.java:591) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:92) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116) at org.eclipse.jetty.server.Server.handle(Server.java:370) at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494) at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971) at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033) at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644) at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235) at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82) at org.eclipse.jetty.io.nio.SslConnection.handle(SslConnection.java:196) at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:696) at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:53) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:748) Locked ownable synchronizers: - None

Answer 1

如下圖在線程轉儲中，id為qtp336276309-556300的線程獲得了org.apache.log4j.spi.RootLogger的鎖並且沒有釋放它。 這會導致其他線程被阻塞。

這是正常的，任何時候只有一個線程可以記錄到一個文件。 如果您有多個線程嘗試同時寫入，則它們必須等待。

這有時伴隨着主機上的 CPU 利用率在一周內從 5% 持續增加到 30%。

雖然日志記錄是導致緩慢的一個非常常見的原因，但它通常不會隨着時間的推移而變得更糟。

通常導致問題的原因是日志記錄過多，隨着時間的推移，您可能會記錄更多日志，這可能會導致增加，但是我會看看是否還有其他原因導致更多活動。 即日志記錄可能只是症狀。

想知道處於阻塞狀態的線程是否會導致 CPU 峰值？

BLOCKED 線程不使用太多 CPU。 雖然這可能會導致更高的延遲，但您可能看不到 CPU 使用率的任何增加（實際上它可能會下降）

如果不是，那么在解決 CPU 使用率飆升的情況下還應該檢查什么？

我會看看其他線程此時正在做什么。 我會嘗試減少日志記錄以減少“噪音”的數量，以便您可以看到大部分時間它在做什么。