How to fix " No managed connections available within configured blocking timeout (60000 [ms]) "

Question

I have deployed java application on wildfly-15.0.1.FINAl server.I am using OJDBC 12.1.0.2.0 driver.

<datasource jndi-name="java:/DS_APP" pool-name="APP" enabled="true" use-java-context="true">
        <connection-url>jdbc:oracle:thin:@localhost:1521:MYDB</connection-url>      
        <driver>OracleJDBCDriver</driver>
        <transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
        <pool>
             <min-pool-size>3</min-pool-size>
             <max-pool-size>200</max-pool-size>
        </pool>
        <security>
             <security-domain>Password4APP</security-domain>
        </security>
        <validation>
            <valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleValidConnectionChecker"/>
            <exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.oracle.OracleExceptionSorter"/>
        </validation>
        <timeout>
            <blocking-timeout-millis>60000</blocking-timeout-millis>
            <idle-timeout-minutes>15</idle-timeout-minutes>
        </timeout>
        <statement>
            <track-statements>true</track-statements>
            <prepared-statement-cache-size>100</prepared-statement-cache-size>
        </statement>
    </datasource>

There was a period when my application could not get database connection and was logging errors like this

Caused by: java.sql.SQLException: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/DS_APP
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:146) ~[?:?]
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64) ~[?:?]
    at ge.app.ws.util.DbManager.getDatabaseConnection(DbManager.java:55) ~[classes:?]
    ... 2 more
Caused by: javax.resource.ResourceException: IJ000453: Unable to get managed connection for java:/DS_APP
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:690) ~[?:?]
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440) ~[?:?]
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789) ~[?:?]
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ~[?:?]
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64) ~[?:?]
    at ge.app.ws.util.DbManager.getDatabaseConnection(DbManager.java:55) ~[classes:?]
    ... 2 more
Caused by: javax.resource.ResourceException: IJ000655: No managed connections available within configured blocking timeout (60000 [ms])
    at org.jboss.jca.core.connectionmanager.pool.mcp.SemaphoreConcurrentLinkedDequeManagedConnectionPool.getConnection(SemaphoreConcurrentLinkedDequeManagedConnectionPool.java:570) ~[?:?]
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getSimpleConnection(AbstractPool.java:632) ~[?:?]
    at org.jboss.jca.core.connectionmanager.pool.AbstractPool.getConnection(AbstractPool.java:604) ~[?:?]
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.getManagedConnection(AbstractConnectionManager.java:624) ~[?:?]
    at org.jboss.jca.core.connectionmanager.tx.TxConnectionManagerImpl.getManagedConnection(TxConnectionManagerImpl.java:440) ~[?:?]
    at org.jboss.jca.core.connectionmanager.AbstractConnectionManager.allocateConnection(AbstractConnectionManager.java:789) ~[?:?]
    at org.jboss.jca.adapters.jdbc.WrapperDataSource.getConnection(WrapperDataSource.java:138) ~[?:?]
    at org.jboss.as.connector.subsystems.datasources.WildFlyDataSource.getConnection(WildFlyDataSource.java:64) ~[?:?]
    at ge.app.ws.util.DbManager.getDatabaseConnection(DbManager.java:55) ~[classes:?]
    ... 2 more

I read many blogs about this error, but my point is different. From blogs I found some problem issues:

1. Maybe you have Connection leak. You are not closing connections
   • What I done: I checked every line of my code and I am sure every Connection and CallableStatement are closing.
2. Maybe all connections from the pool are already in use, Try to grow your connection pool size which is default 20
   • What I done: When there was this problem, I checked connections on database side and there was no active connection, all connections were inactive, also I have 200 connections in pool which is so many, I have maximum 30 active session at the same time and I have 60 Second time out to wait until connections from pool is free.

When I restarted my Wildfly server the problem solved, but I am interested in why this errors happened, if there can be this errors in the future(I think they will occur again) and how to avoid them?

Answer 1

I faced the same issue recently. And as you mentioned, the possible solutions for this fix are to deal with

1. Datasource configuration
2. Leaks while closing the connections.

I tried changing the datasource configuration by setting max pool size to 20 like that, But i got the same error after hitting 21st request.

I enabled stats in datasource configuration and verified the pool statistics in jboss console->datasource configuration and found that connections are getting created but not getting closed.

You will not get this error once you restart it because all these connections would get flushed and 20 new connections will be readily available to serve.

So i tried to track the connection leaks and added these below configurations which would manually close the datasource connection and logs an error if there is a connection leak . https://access.redhat.com/solutions/309913

It solved my issue but this cannot be the permanent solution so rechecked the code to see where the connection got leaked. Then i changed the code to properly close the datasource.

I thought this would work but it didn't (Initial code):

   try {
         Connection dbConnect = dataSource.getConnection();
         Statement stmt = dbConnect.createStatement();
         ResultSet rs = stmt.executeQuery("select s from soething");
         stmt.close();
         dbConnect.close();
         return something;
    } catch (SQLException e) {
         LOGGER.log(Level.INFO, e.getMessage());
         return e.getMessage();
    }

This would work (Final code with neatly closed datasources):

        Statement stmt=null;
        Connection dbConnect=null;
          try {
                dbConnect = dataSource.getConnection();
                stmt = dbConnect.createStatement();
                ResultSet rs = stmt.executeQuery("select s from soething");
                return something;
            } catch (SQLException e) {
                LOGGER.log(Level.INFO, e.getMessage());
                return e.getMessage();
            }finally{
                if (stmt != null) stmt.close();
                if (dbConnect != null) dbConnect.close();
            }