RabbitMQ影响JDBC连接池

Question

I have a web servlet that has been running well. I decided to replace a TCP connection to a backend server with RabbitMQ. While doing some load testing I noticed that it quickly started to fail with timeouts waiting for idle JDBC connection.

ERROR [http-bio-8080-exec-12] JDBCExceptionReporter.logExceptions(234) | Cannot get a connection, pool error Timeout waiting for idle object
DEBUG [http-bio-8080-exec-12] JDBCExceptionReporter.logExceptions(225) | Cannot open connection [???]
org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
    at org.springframework.orm.hibernate3.LocalDataSourceConnectionProvider.getConnection(LocalDataSourceConnectionProvider.java:85)
    at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
    at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
    at org.hibernate.jdbc.JDBCContext.connection(JDBCContext.java:160)
    at org.hibernate.transaction.JDBCTransaction.begin(JDBCTransaction.java:81)
    at org.hibernate.impl.SessionImpl.beginTransaction(SessionImpl.java:1473)
    at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:560)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:438)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:261)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at net.hedtech.degreeworks.security.service.ShpPassportService$$EnhancerBySpringCGLIB$$9c441f05.getAndCheckPassportForRequest(<generated>)
    at net.hedtech.degreeworks.security.spring.ShpPassportFilter.doFilter(ShpPassportFilter.java:124)
    at net.hedtech.degreeworks.security.spring.ShpPassportFilter$$FastClassBySpringCGLIB$$8067e598.invoke(<generated>)
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
    at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:717)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
    at com.sungardhe.degreeworks.util.SpringBeansProfiler.aroundAdvice(SpringBeansProfiler.java:92)
    at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:653)
    at net.hedtech.degreeworks.security.spring.ShpPassportFilter$$EnhancerBySpringCGLIB$$c08501d.doFilter(<generated>)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:50)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
    at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
    at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
    at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:344)
    at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:261)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:504)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:421)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1074)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:611)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)

I have not implemented any pooling myself at this point. The RabbitMQ connection is a singleton shared between all the session threads. A channel is opened and closed for each request.

My research has not turned up any connection between RabbitMQ and jdbc connection pools, but I have to admit that I only have a passing knowledge of either.

Here is the code I am using:

String requestQueueName = "dw-dev-prototype";

QueueingConsumer consumer;

Channel channel = null;
String correlationId = UUID.randomUUID().toString();
String thisResponse = null;
String totalResponse = "";
StringBuffer sbResponse;
// We will create a new reply queue for each request
String replyQueueName = "";

try
{
   // We will send the replyQueueName with the request so the server knows how to send the
   // response back to us
   channel = _connection.createChannel();
   replyQueueName = channel.queueDeclare().getQueue();
   consumer = new QueueingConsumer(channel);
   channel.basicConsume(replyQueueName, AUTO_ACKNOWLEDGE, consumer);
   BasicProperties props = new BasicProperties.Builder().correlationId(correlationId).replyTo(replyQueueName)
      .build();
   channel.basicPublish(NO_EXCHANGE, requestQueueName, props, sNameValuePairs.getBytes("UTF-8"));
}
catch (IOException e)
{
   log.error("New connection IOException: " + e.getMessage(), e);
   abortConnection(_connection, channel);
   throw new RuntimeException(e);
}

while (true)
{
   QueueingConsumer.Delivery delivery;
   try
   {
      // Get the next response
      delivery = consumer.nextDelivery();
      thisResponse = new String(delivery.getBody(), "UTF-8");
      // Ignore the response if it is not for our correlation-id
      else if (delivery.getProperties().getCorrelationId().equals(correlationId))
      {
         if (thisResponse.equals(this._serviceFinishedMsg))
         {
            // No more text for this response - we are done
            break;
         }
         else
         {
            totalResponse += thisResponse;
         }
      }
   }
   catch (ShutdownSignalException | ConsumerCancelledException | InterruptedException e)
   {
      log.error("Consumer exception: " + e.getMessage(), e);
      throw new RuntimeException(e);
   }
   catch (UnsupportedEncodingException e)
   {
      log.error("UnsupportedEncoding exception: " + e.getMessage(), e);
      throw new RuntimeException(e);
   }

}

// Close now that we are done
try
{
   if (channel != null && channel.isOpen())
   {
      channel.close();
   }
}
catch (IOException | TimeoutException e)
{
   log.error("Error closing channel or connection", e);
}

I am running this in a Tomcat container. I am using Spring, but not Spring AMQP yet.

The JDBC calls are in a security filter that runs before the request gets to this method.

Any hints as to how I might troubleshoot this would be welcome.

Thanks in advance.

Answer 1

A call that you initiated invoked aop -> spring service -> hibernate. Hibernate is invoking commons dbcp and asking for a connection pool. You're likely running out of connections from this pool or the db has become unreachable. I'd vote for the former.

Integration java melody into your app and deploy it again. Monitor the connection pool and check if you run out. If you do...

• Consider increasing the connection pool limit
• If increasing connection pool limit does not help, you might have a dbcp pool leak. Investigate if you are returning connections back to the pool correctly.