Why Throwable::printStackTrace holding lock of PrintStream and cause deadlock of logback

Question

Found a deadlock situation when using e.printStackTrace() and logback in different threads. The thread dumps are given below.

It seems to me, the logback (used in thread AsyncAppender-Worker-Thread-1 ) trying to acquire the lock of PrintStream , which is already owned by by main thread 's java.lang.Throwable$WrappedPrintStream.println . If that's the case, why the printStackTrace keep holding the lock of PrintStream (as it should release it once the printing is done)?

Thread dump For the main thread .

"main@1" prio=5 tid=0x1 nid=NA waiting
  java.lang.Thread.State: WAITING
     blocks AsyncAppender-Worker-Thread-1@831
      at sun.misc.Unsafe.park(Unsafe.java:-1)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      at java.util.concurrent.ArrayBlockingQueue.put(ArrayBlockingQueue.java:353)
      at ch.qos.logback.core.AsyncAppenderBase.put(AsyncAppenderBase.java:139)
      at ch.qos.logback.core.AsyncAppenderBase.append(AsyncAppenderBase.java:130)
      at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
      at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
      at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:273)
      at ch.qos.logback.classic.Logger.callAppenders(Logger.java:260)
      at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:442)
      at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
      at ch.qos.logback.classic.Logger.error(Logger.java:543)
      at com.side.stdlib.logging.StdOutErrLog$2.print(StdOutErrLog.java:43)
      at java.io.PrintStream.println(PrintStream.java:823)
      - locked <0x1183> (a com.side.stdlib.logging.StdOutErrLog$2)
      at java.lang.Throwable$WrappedPrintStream.println(Throwable.java:749)
      at java.lang.Throwable.printEnclosedStackTrace(Throwable.java:698)
      at java.lang.Throwable.printStackTrace(Throwable.java:668)
      at java.lang.Throwable.printStackTrace(Throwable.java:644)
      at java.lang.Throwable.printStackTrace(Throwable.java:635)
      at com.side.SidekApi.sideAPIExecution(SidekApi.java:175)

Thread dump for the thread AsyncAppender-Worker-Thread-1

"AsyncAppender-Worker-Thread-1@831" daemon prio=5 tid=0xe nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
     waiting for main@1 to release lock on <0x1183> (a com.side.stdlib.logging.StdOutErrLog$2)
      at java.io.PrintStream.write(PrintStream.java:478)
      at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
      at ch.qos.logback.core.joran.spi.ConsoleTarget$2.write(ConsoleTarget.java:55)
      at ch.qos.logback.core.encoder.LayoutWrappingEncoder.doEncode(LayoutWrappingEncoder.java:135)
      at ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:194)
      at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:219)
      at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:103)
      at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:88)
      at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:48)
      at ch.qos.logback.core.AsyncAppenderBase$Worker.run(AsyncAppenderBase.java:226)

It seems the situation is a bit similar with https://bugs.openjdk.java.net/browse/JDK-6719464 , but no answer there.

Answer 1

If the logback worker thread can't finish, it must be because its blocking queue is full. The worker is waiting to deposit its log entry, and since the thread is WAITING we know it released the lock on the queue, but it still holds the lock on the printstream. The console writing thread is BLOCKED trying to acquire the lock on the printstream, which it needs in order to write to the console, so they are deadlocked.

A minimal fix that avoids code changes could be swapping out the console appender for one that doesn't need to acquire a lock on the printstream.

In any case needing to take the lock on the printstream probably reduces the benefit from logging asynchronously. The long term fix will involve replacing the printlns with calls to a logger (like slf4j).