tomcat：在条件线程上等待

Question

I'm investigating the strange issue with tomcat shutdown process: after runnig shutdown.sh the java process still appears(I check it by using ps -ef|grep tomcat ) The situation is a bit complicated, because I have very limitted access to the server(no debug, for example)

I took thread dump(by using kill -3 <PID> ) and heap dump by using remote jConsole and Hotspot features.

After looking into thread dump I found this:

"pool-2-thread-1" #74 prio=5 os_prio=0 tid=0x00007f3354359800 nid=0x7b46 waiting on condition [0x00007f333e55d000]
  java.lang.Thread.State: WAITING (parking)
       at sun.misc.Unsafe.park(Native Method)
       - parking to wait for  <0x00000000c378d330> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
       at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
       at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
       at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
       at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
       at java.lang.Thread.run(Thread.java:748)

So, my understanding of the problem is follows: There is a resource(DB connection or something else) which is used in CachedThreadpool, and this resource is now locked, and prevent to thread pool-2-thread-1 to stop. Assuming that this thread isn't deamon - JVM cannot gracefully stop.

Is there a way to find out which resource is locked, from where is it locked and how to avoid that? Another question is - how to prevent this situation? Another this is: what the adress 0x00007f333e55d000 is for?

Thanks!

Answer 1

After a wail of struggling I found out, that the freezing thread always have the same name pool-2-thread-1 . I grep all the sources of the project to find any places where any scheduled thread pool is started: Executors#newScheduledThreadPool . After a huge amount of time I logged all that places, with thread name in thread pool. One more restart server and gotcha!

I found out one thread pool, that is started with only one thread and used following code:

private void shutdownThreadpool(int tries) {
    try {
        boolean terminated;
        LOGGER.debug("Trying to shutdown thread pool in {} tries", tries);
        pool.shutdown();
        do {
            terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN,TimeUnit.MILLISECONDS);
            if (--tries == 0
                    && !terminated) {
                LOGGER.debug("After 10 attempts couldn't shutdown thread pool, force shutdown");
                pool.shutdownNow();
                terminated = pool.awaitTermination(WAIT_ON_SHUTDOWN, TimeUnit.MILLISECONDS);
                if (!terminated) {
                    LOGGER.debug("Cannot stop thread pool even with force");
                    LOGGER.trace("Some of the workers doesn't react to Interruption event properly");
                    terminated = true;
                }
            } else {
                LOGGER.info("After {} attempts doesn't stop", tries);
            }
        } while (!terminated);
        LOGGER.debug("Successfully stop thread pool");
    } catch (final InterruptedException ie) {
        LOGGER.warn("Thread pool shutdown interrupted");
    }
}

After that the issue was solved.