spring-boot 关机兔子 NoClassDefFoundError

Question

Occasionally NoClassDefFoundError is prompted to quit the JVM process, causing the JVM process to quit normally.

https://github.com/spring-projects/spring-amqp/issues/822

I use:

compile "org.springframework.boot:spring-boot-starter-amqp:2.0.5.RELEASE"

error1:

2018-10-15 11:16:27,350 [Thread-13] [] [WARN ] o.s.c.s.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:264) - Failed to stop bean 'org.springframework.amqp.rabbit.config.internalRabbitListenerEndpointRegistry'
java.lang.NoClassDefFoundError: org/springframework/amqp/rabbit/listener/RabbitListenerEndpointRegistry$AggregatingCallback
        at org.springframework.amqp.rabbit.listener.RabbitListenerEndpointRegistry.stop(RabbitListenerEndpointRegistry.java:260)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:238)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:53)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:377)
        at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:210)
        at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:128)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1005)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933)
Caused by: java.lang.ClassNotFoundException: org.springframework.amqp.rabbit.listener.RabbitListenerEndpointRegistry$AggregatingCallback
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 8 common frames omitted

error2:

2018-10-15 11:16:57,353 [Thread-13] [] [INFO ] o.s.c.s.DefaultLifecycleProcessor.stop(DefaultLifecycleProcessor.java:387) - Failed to shut down 1 bean with phase value 2147483647 within timeout of 30000: [org.springframework.amqp.rabbit.config.internalRabbitListenerEndpointRegistry]
2018-10-15 11:16:57,358 [Thread-13] [] [WARN ] o.s.c.s.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:264) - Failed to stop bean 'rabbitTemplate'
java.lang.BootstrapMethodError: java.lang.NoClassDefFoundError: org/springframework/amqp/rabbit/listener/DirectReplyToMessageListenerContainer
        at org.springframework.amqp.rabbit.core.RabbitTemplate.isRunning(RabbitTemplate.java:791)
        at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:231)
        at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:53)
        at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:377)
        at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:210)
        at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:128)
        at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1005)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933)
Caused by: java.lang.NoClassDefFoundError: org/springframework/amqp/rabbit/listener/DirectReplyToMessageListenerContainer
        ... 8 common frames omitted
Caused by: java.lang.ClassNotFoundException: org.springframework.amqp.rabbit.listener.DirectReplyToMessageListenerContainer
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 8 common frames omitted
2018-10-15 11:16:57,364 [AMQP Connection 192.168.31.231:5672] [] [ERROR] c.r.c.i.ForgivingExceptionHandler.log(ForgivingExceptionHandler.java:119) - An unexpected connection driver error occured
java.lang.NoClassDefFoundError: com/rabbitmq/client/impl/ConsumerDispatcher$2
        at com.rabbitmq.client.impl.ConsumerDispatcher.handleCancelOk(ConsumerDispatcher.java:91)
        at com.rabbitmq.client.impl.ChannelN$7.transformReply(ChannelN.java:1464)
        at com.rabbitmq.client.impl.ChannelN$7.transformReply(ChannelN.java:1458)
        at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.handleCommand(AMQChannel.java:478)
        at com.rabbitmq.client.impl.RpcContinuationRpcWrapper.complete(RpcContinuationRpcWrapper.java:38)
        at com.rabbitmq.client.impl.AMQChannel.handleCompleteInboundCommand(AMQChannel.java:196)
        at com.rabbitmq.client.impl.AMQChannel.handleFrame(AMQChannel.java:111)
        at com.rabbitmq.client.impl.AMQConnection.readFrame(AMQConnection.java:643)
        at com.rabbitmq.client.impl.AMQConnection.access$300(AMQConnection.java:47)
        at com.rabbitmq.client.impl.AMQConnection$MainLoop.run(AMQConnection.java:581)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: com.rabbitmq.client.impl.ConsumerDispatcher$2
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at org.springframework.boot.loader.LaunchedURLClassLoader.loadClass(LaunchedURLClassLoader.java:93)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 11 common frames omitted

thread dump:

"Thread-13" #45 prio=5 os_prio=0 tid=0x00007f000c024800 nid=0x108b in Object.wait() [0x00007effcd4e4000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00000000e779bf50> (a com.rabbitmq.utility.BlockingValueOrException)
    at com.rabbitmq.utility.BlockingCell.get(BlockingCell.java:73)
    - locked <0x00000000e779bf50> (a com.rabbitmq.utility.BlockingValueOrException)
    at com.rabbitmq.utility.BlockingCell.uninterruptibleGet(BlockingCell.java:120)
    - locked <0x00000000e779bf50> (a com.rabbitmq.utility.BlockingValueOrException)
    at com.rabbitmq.utility.BlockingValueOrException.uninterruptibleGetValue(BlockingValueOrException.java:36)
    at com.rabbitmq.client.impl.AMQChannel$BlockingRpcContinuation.getReply(AMQChannel.java:494)
    at com.rabbitmq.client.impl.ChannelN.basicCancel(ChannelN.java:1477)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.amqp.rabbit.connection.CachingConnectionFactory$CachedChannelInvocationHandler.invoke(CachingConnectionFactory.java:1032)
    - locked <0x0000000088906598> (a java.lang.Object)
    at com.sun.proxy.$Proxy84.basicCancel(Unknown Source)
    at org.springframework.amqp.rabbit.listener.BlockingQueueConsumer.basicCancel(BlockingQueueConsumer.java:399)
    at org.springframework.amqp.rabbit.listener.SimpleMessageListenerContainer.doShutdown(SimpleMessageListenerContainer.java:521)
    - locked <0x0000000088906a60> (a java.lang.Object)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.shutdown(AbstractMessageListenerContainer.java:1100)
    at org.springframework.amqp.rabbit.listener.AbstractMessageListenerContainer.destroy(AbstractMessageListenerContainer.java:1042)
    at org.springframework.amqp.rabbit.listener.RabbitListenerEndpointRegistry.destroy(RabbitListenerEndpointRegistry.java:221)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:256)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:571)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:543)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1054)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:504)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1061)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1037)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1013)
    at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933)
    - locked <0x00000000879b14a8> (a java.lang.Object)

Rabbit.MainLoopThread has exited, but SimpleMessageListenerContainer is still waiting for a message to respond. In fact, it will never receive a message. This thread prevents the JVM process from quitting properly.

I don't know why NoClassDefFoundError is prompted here. I'm sure these classes exist, and it's not necessarily possible. I haven't found a reproducing method yet. Initially I used ampq-client-4.0.x, which I thought was too low, and I've upgraded spring-boot-2.x and ampq-client-5.x, but there's the same problem. I really feel helpless. Please give me some hints.

This is my dependence. spring-boot-starter-amqp is the default. I didn't modify it.

\--- org.springframework.boot:spring-boot-starter-amqp:2.0.5.RELEASE
     +--- org.springframework.boot:spring-boot-starter:2.0.5.RELEASE (*)
     +--- org.springframework:spring-messaging:5.0.9.RELEASE
     |    +--- org.springframework:spring-beans:5.0.9.RELEASE -> 5.1.0.RELEASE (*)
     |    \--- org.springframework:spring-core:5.0.9.RELEASE -> 5.1.0.RELEASE (*)
     \--- org.springframework.amqp:spring-rabbit:2.0.6.RELEASE
          +--- org.springframework.amqp:spring-amqp:2.0.6.RELEASE
          |    +--- org.springframework:spring-core:5.0.9.RELEASE -> 5.1.0.RELEASE (*)
          |    \--- org.springframework.retry:spring-retry:1.2.2.RELEASE
          |         \--- org.springframework:spring-core:4.3.13.RELEASE -> 5.1.0.RELEASE (*)
          +--- com.rabbitmq:amqp-client:5.1.2
          |    \--- org.slf4j:slf4j-api:1.7.25
          +--- org.springframework:spring-context:5.0.9.RELEASE -> 5.1.0.RELEASE (*)
          +--- org.springframework:spring-messaging:5.0.9.RELEASE (*)
          +--- org.springframework:spring-tx:5.0.9.RELEASE -> 5.1.0.RELEASE (*)
          \--- org.springframework:spring-web:5.0.9.RELEASE -> 5.1.0.RELEASE (*)

This is my dependence. spring-boot-starter-amqp is the default. I didn't modify it.

Answer 1

if you are using newrelic agent, desable spring oap instrumentation in newrelic config. Ex: com.newrelic.instrumentation.spring-aop-2: enabled: false