重载下Java进程不会在SIGTERM上退出

Question

In normal operation my application exits fine when sent a 'kill -s SIGTERM '.

However, under load sometimes the process does not exit.

I'm just wondering if it possible that http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6392332 is the reason for this, or if it could be something else?

Here are some parts of the stack trace of the process in question, showing the shutdown methods, any help is much appreciated.

Note, this is a Java process running on 64-bit RHEL 6.3.

2013-05-22 08:01:33 
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.21-b01 mixed mode): 

...

"Thread-15" prio=10 tid=0x000000001994d000 nid=0x4d5a waiting on condition [0x00007f4da08a3000] 
   java.lang.Thread.State: TIMED_WAITING (parking) 
at sun.misc.Unsafe.park(Native Method) 
- parking to wait for <0x000000079fd9fce8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) 
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226) 
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082) 
at java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468) 
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:109) 
at org.jboss.netty.util.internal.ExecutorUtil.terminate(ExecutorUtil.java:49) 
at org.jboss.netty.channel.socket.nio.AbstractNioWorkerPool.releaseExternalResources(AbstractNioWorkerPool.java:77) 
at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:164) 
at com.test.services.radius.server.RadiusServerImpl.stop(RadiusServerImpl.java:87) 
at com.test.services.radius.ServiceProvider.unload(ServiceProvider.java:61) 
at com.test.spf.ServiceProviderCacheImpl.clearCurrentCache(ServiceProviderCacheImpl.java:150) 
- locked <0x00000006b8968038> (a com.test.spf.ServiceProviderCacheImpl) 
at com.test.spf.ServiceProviderCacheImpl.unload(ServiceProviderCacheImpl.java:170) 
at com.test.spf.SPAImpl.stop(SPAImpl.java:178) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:601) 
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:273) 
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:199) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:487) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:480) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:463) 
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:431) 
- locked <0x00000006da966290> (a java.util.LinkedHashMap) 
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1048) 
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1022) 
at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:970) 
- locked <0x000000073ad1a920> (a java.lang.Object) 
at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:384) 
at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:78) 
at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4245) 
at org.apache.catalina.core.StandardContext.stop(StandardContext.java:4886) 
- locked <0x00000006b8968118> (a org.apache.catalina.core.StandardContext) 
at org.apache.catalina.core.ContainerBase.removeChild(ContainerBase.java:936) 
at org.apache.catalina.startup.HostConfig.undeployApps(HostConfig.java:1359) 
at org.apache.catalina.startup.HostConfig.stop(HostConfig.java:1330) 
at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:326) 
at org.apache.catalina.util.LifecycleSupport.fireLifecycleEvent(LifecycleSupport.java:142) 
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1098) 
- locked <0x00000006b89682f8> (a org.apache.catalina.core.StandardHost) 
at org.apache.catalina.core.ContainerBase.stop(ContainerBase.java:1110) 
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine) 
at org.apache.catalina.core.StandardEngine.stop(StandardEngine.java:468) 
at org.apache.catalina.core.StandardService.stop(StandardService.java:604) 
- locked <0x000000068e63ff10> (a org.apache.catalina.core.StandardEngine) 
at org.apache.catalina.core.StandardServer.stop(StandardServer.java:788) 
at org.apache.catalina.startup.Catalina.stop(Catalina.java:662) 
at org.apache.catalina.startup.Catalina$CatalinaShutdownHook.run(Catalina.java:706) 

"SIGTERM handler" daemon prio=10 tid=0x0000000025453000 nid=0x4d58 in Object.wait() [0x00007f4da0b2f000] 
   java.lang.Thread.State: WAITING (on object monitor) 
at java.lang.Object.wait(Native Method) 
- waiting on <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook) 
at java.lang.Thread.join(Thread.java:1258) 
- locked <0x000000072c10db20> (a org.apache.catalina.startup.Catalina$CatalinaShutdownHook) 
at java.lang.Thread.join(Thread.java:1332) 
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106) 
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46) 
at java.lang.Shutdown.runHooks(Shutdown.java:123) 
at java.lang.Shutdown.sequence(Shutdown.java:167) 
at java.lang.Shutdown.exit(Shutdown.java:212) 
- locked <0x0000000707855738> (a java.lang.Class for java.lang.Shutdown) 
at java.lang.Terminator$1.handle(Terminator.java:52) 
at sun.misc.Signal$1.run(Signal.java:212) 
at java.lang.Thread.run(Thread.java:722) 

Answer 1

No Unix process will finish on SIGTERM till all threads are finished properly, thus returned from run method . High load could have its roots in deadlock - deadlocked threads usually never ends. Or endless loop in some thread.

Btw proper way of shutting down the Tomcat is to use bundled shutdown script. It could fail in some extraordinary situations, but then you can just SIGKILL it.

SIGTERM or SIGKILL, that's often the business question. Proper shutdown can easily take more than 15min, when complex application, swapped... aso So can you stand 15min outage or rather you'll kill it and start in next 2min?