Java / Hibernate in client with reconnect

Question

I have a mysql server running 24/7 and a lot of SWT rich clients accessing the database using JPA / Hibernate...

If the client computers (laptops) go to hibernate / standby and wake up some time later the database connection got lost - no matter what kind of setting and / or database-connection-provider (I tried c3p0 and vibur) I use....

Here's my current hibernate.xml properties file (which also does not work)...:

    <property name="connection.pool_size">3</property>
    <property name="cache.provider_class">org.hibernate.cache.EhCacheProvider</property>
    <property name="hibernate.cache.use_query_cache">false</property>

    <property name="hibernate.current_session_context_class">thread</property>

    <property name="hibernate.c3p0.validate">true</property>
    <property name="hibernate.connection.provider_class">org.hibernate.service.jdbc.connections.internal.C3P0ConnectionProvider</property>
    <property name="hibernate.c3p0.min_size">5</property>
    <property name="hibernate.c3p0.max_size">600</property>
    <property name="hibernate.c3p0.timeout">0</property>
    <property name="hibernate.c3p0.max_statements">50</property>
    <property name="hibernate.c3p0.preferredTestQuery">SELECT 1;</property>
    <property name="hibernate.c3p0.testConnectionOnCheckout">true</property>
    <property name="hibernate.c3p0.testConnectionOnCheckin">true</property>
            <property name="hibernate.c3p0.idle_test_period">3000</property>
    <property name="hibernate.c3p0.acquireRetryAttempts">10</property>
    <property name="hibernate.c3p0.acquireRetryDelay">1000</property>
    <property name="hibernate.c3p0.breakAfterAcquireFailure">false</property>

And here is an error stack trace:

WARN | 02.12.2014 21:35:06,503 | [C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-AdminTaskTimer] | ThreadPoolAsynchronousRunner:743 | com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@89789 -- APPARENT DEADLOCK!!! Creating emergency threads for unassigned pending tasks!
 WARN | 02.12.2014 21:35:06,507 | [C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-AdminTaskTimer] | ThreadPoolAsynchronousRunner:759 | com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockDetector@89789 -- APPARENT DEADLOCK!!! Complete Status: 
    Managed Threads: 3
    Active Threads: 3
    Active Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@b76a87
            on thread: C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#0
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@1b2ffe7
            on thread: C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#1
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@becbe1
            on thread: C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#2
    Pending Tasks: 
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@3b2d78
Pool thread stack traces:
    Thread[C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#0,5,main]
        java.net.DualStackPlainSocketImpl.connect0(Native Method)
        java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
        java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
        java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        java.net.Socket.connect(Socket.java:579)
        java.net.Socket.connect(Socket.java:528)
        java.net.Socket.<init>(Socket.java:425)
        java.net.Socket.<init>(Socket.java:241)
        com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
        com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:307)
        com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2482)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        sun.reflect.GeneratedConstructorAccessor249.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
        com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
    Thread[C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#2,5,main]
        java.net.DualStackPlainSocketImpl.connect0(Native Method)
        java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
        java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
        java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        java.net.Socket.connect(Socket.java:579)
        java.net.Socket.connect(Socket.java:528)
        java.net.Socket.<init>(Socket.java:425)
        java.net.Socket.<init>(Socket.java:241)
        com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
        com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:307)
        com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2482)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        sun.reflect.GeneratedConstructorAccessor249.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
        com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
    Thread[C3P0PooledConnectionPoolManager[identityToken->1hge13a96372xk9xvyjyj|1dd3180]-HelperThread-#1,5,main]
        java.net.DualStackPlainSocketImpl.connect0(Native Method)
        java.net.DualStackPlainSocketImpl.socketConnect(DualStackPlainSocketImpl.java:79)
        java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        java.net.PlainSocketImpl.connect(PlainSocketImpl.java:172)
        java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        java.net.Socket.connect(Socket.java:579)
        java.net.Socket.connect(Socket.java:528)
        java.net.Socket.<init>(Socket.java:425)
        java.net.Socket.<init>(Socket.java:241)
        com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFactory.java:259)
        com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:307)
        com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2482)
        com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2519)
        com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2304)
        com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
        com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
        sun.reflect.GeneratedConstructorAccessor249.newInstance(Unknown Source)
        sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
        com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:346)
        com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
        com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
        com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
        com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
        com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
        com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
        com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)


 WARN | 03.12.2014 10:29:20,014 | [main] | SqlExceptionHelper:145 | SQL Error: 0, SQLState: 08S01
ERROR | 03.12.2014 10:29:20,015 | [main] | SqlExceptionHelper:147 | The last packet successfully received from the server was 65.896.659 milliseconds ago.  The last packet sent successfully to the server was 65.896.672 milliseconds ago. is longer than the server configured value of 'wait_timeout'. You should consider either expiring and/or testing connection validity before use in your application, increasing the server configured values for client timeouts, or using the Connector/J connection property 'autoReconnect=true' to avoid this problem.

Answer 1

Threads that were in the process of acquiring new database Connections when the db server (or laptop client) went down have gotten stuck. They are midcommunucation with the DBMS and simply hanging. They are neither successfully acquiring a Connection, nor failing promptly with an Exception. These hung Threads are saturating your Thread pool, preventing a variety of tasks from being accomplished. After some time, c3p0 declares an APPARENT DEADLOCK!, then discards and replaces all the Threads in its Thread pool to start over. Usually application recover and function normally following an APPARENT DEADLOCK! But, if Connection attempts interrupted by an outage neither succeed nor fail with an Exception, you might occasionally see these deadlocks.

A few things might help a bit. If the I/O operations in which Connection acquisition attempts are stuck are interruptable,than you may be able to prevent the APPARENT DEADLOCKS by setting a maxAdministrativeTaskTime . But if the I/O is not interruptible, that won't work.

If you are seeing these deadlocks fairly regularly, that probably means you are churning through Connections too quickly. Connection acquisition attempts should be pretty infrequent in a well-tuned pool, so having an outage occur mid-multiple-acquisitions should also be pretty rare. There's nothing in the config above that should cause you to churn through Connections, but you say that the config you've quoted "doesn't work", so maybe the real c3p0 config is different. c3p0 DataSource's dump its config on pool init, at INFO level. If you search your logs, you can verify whether the config you intend is actually the config of your c3p0 DataSource.

Answer 2

Since they are all hanging on connect, try setting a connectTimeout for the MySQL JDBC driver (mentioned in the MySQL connector J documentation ). By default, there is no timeout which could leave them hanging forever.

Going to standby and waking up is something you should catch in your client application so that you can take action to prevent errors (eg close the database connections and then open new ones). See this answer about how you can catch some of these kind of events, unfortunately the answer does not include an example with the WM_POWERBROADCAST event (on Windows) which appears to signal standby/wake-up. I did find an article about a StandByDetector for Windows (YMMV).

While you are at it, you might want to try the HikariCP database connection pool instead of c3p0 (I suspect HikariCP is better at recovering from broken database connections). You can browse the setup-demo HibHik to get started.

Answer 3

Try HikariCP. See our DB-down torture test . This only works with a good modern driver that supports Connection.setNetworkTimeout().