Losing JPA connection after some idle time

Question

I do lose the connection to my database every time the API is idle for some time (the time cannot be defined precisely. It can happen after idle time of just some seconds, sometimes idle time after minutes, sometimes even right after jetty restart). This issue is reproducible on different machines, so I recon it has nothing to do with the mysql database directly.

The exception thrown can be found here

The database configuration is based upon hibernate and spring:

 <!-- c3p0 of version 0.9.2.1 (current stable) -->
 <!-- spring data: 1.2.0.RELEASE -->
 <!-- mysql connector: 5.1.18 -->
 <!-- hibernate: 4.1.9.FINAL --> 
 <bean id="basisDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
    <property name="driverClass" value="${jdbc.driverClassName}" />
    <property name="jdbcUrl" value="${jdbc.url}" />
    <property name="user" value="${jdbc.username}" />
    <property name="password" value="${jdbc.password}" />
    <property name="minPoolSize" value="5" />
    <property name="maxPoolSize" value="20" />
    <property name="maxIdleTime" value="600" />
    <property name="maxConnectionAge" value="86400" />
</bean>

<bean id="entityManagerFactory"
    class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="packagesToScan" value="de.yourdelivery.data.*" />
    <property name="jpaVendorAdapter">
        <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
            <property name="generateDdl" value="false" />
            <property name="showSql" value="true" />
            <property name="database" value="MYSQL" />
        </bean>
    </property
</bean>

<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
    <property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>

the Database connection and driver is a followed:

jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://xxx:3306/xxx?zeroDateTimeBehavior=convertToNull

it does not even work with extending the jdbc.url by

&autoReconnect=true&autoReconnectForPools=true

I use OpenSessionInView Pattern to lazy load all my needed relations during dozer mapping (I know it is deemed to be an anti pattern, but it worked fine so far so currently not blaming it on that pattern)

Mysql wait_timeout is set to 28800

Startup log for C3PO looks like that

13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 - Loading properties file from file [/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/customer.fidelity.points.properties] 13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 - Loading properties file from file [/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/application.properties] 13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 - Loading properties file from file [/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/restapi.properties] 13:34:54,688 INFO PropertySourcesPlaceholderConfigurer:177 - Loading properties file from file [/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/database.properties] 13:34:54,823 INFO MLog:80 - MLog clients using log4j logging. 13:34:54,883 INFO C3P0Registry:204 - Initializing c3p0-0.9.1.2 [built 21-May-2007 15:04:56; debug? true; trace: 10] 13:34:55,111 INFO XmlWebApplicationContext:1350 - Bean 'deDataSource' of type [class com.mchange.v2.c3p0.ComboPooledDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 13:34:55,148 INFO XmlWebApplicationContext:1350 - Bean 'plDataSource' of type [class com.mchange.v2.c3p0.ComboPooledDataSource] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 13:34:55,150 INFO XmlWebApplicationContext:1350 - Bean 'dataSource' of type [class de.yourdelivery.data.routing.DomainRouting] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying) 13:34:55,167 INFO XmlWebApplicationContext:1350 - Bean 'org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter#24bb6086' of type [class org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not eligible for getting processed by all BeanPostProcessors (for exa mple: not eligible for auto-proxying) 13:34:55,213 INFO LocalContainerEntityManagerFactoryBean:264 - Building JPA container EntityManagerFactory for persistence unit 'default' 13:34:55,332 INFO Version:37 - HCANN000001: Hibernate Commons Annotations {4.0.1.Final} 13:34:55,338 INFO Version:41 - HHH000412: Hibernate Core {4.1.9.Final} 13:34:55,341 INFO Environment:239 - HHH000206: hibernate.properties not found 13:34:55,342 INFO Environment:342 - HHH000021: Bytecode provider name : javassist 13:34:55,361 INFO Ejb3Configuration:527 - HHH000204: Processing PersistenceUnitInfo [ name: default ...] 13:34:55,795 INFO ConnectionProviderInitiator:188 - HHH000130: Instantiating explicit connection provider: org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider 13:34:55,880 INFO AbstractPoolBackedDataSource:462 - Initializing c3p0 pool... com.mchange.v2.c3p0.ComboPooledDataSource [ acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay -> 1000, autoCommitOnClose -> false , automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0, connectionCustomizerClassName -> null, connectionTesterClassName -> com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName -> 1hge17b8s1e4h6uz10tbcor|3efe0ce9, debugUnreturnedConnectionStackTraces -> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions -> false, identityToken -> 1hge17b8s1e4h6uz10tbcor|3efe0ce9, idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl -> jdbc:mysql://192.168.1.10:3306/xxxx?zeroDateTimeBehavior=convertToNull, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 86400, maxIdleTime -> 600, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5, numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0, preferredTestQuery -> null, properties -> {user= * , password= * }, propertyCycle -> 0, testConnectionOnCheckin -> false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout -> 0, usesTraditionalReflectiveProxies -> false ] 13:34:56,086 INFO Dialect:123 - HHH000400: Using dialect: org.hibernate.dialect.MySQLDialect 13:34:56,095 INFO LobCreatorBuilder:120 - HHH000424: Disabling contextual LOB creation as createClob() method threw error : java.lang.reflect.InvocationTargetException 13:34:56,108 INFO TransactionFactoryInitiator:73 - HHH000268: Transaction strategy: org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory 13:34:56,112 INFO ASTQueryTranslatorFactory:48 - HHH000397: Using ASTQueryTranslatorFactory 13:34:56,149 INFO Version:24 - Hibernate Validator 4.2.0.Final 13:34:56,916 INFO XmlWebApplicationContext:1350 - Bean 'entityManagerFactory' of type [class org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for au to-proxying)

So why is the application losing its connection during idle time?

Answer 1

Interesting problem

It can happen due to possibly multiple reasons. your app server is losing connections to your DB. It is hard to tell looking at the logs that only connections which are idle are the only one which are getting killed. Your DB server might be not allocating connections. I hope you have checked your DB logs for errors. How you are handling transactions in your services ? Are you using spring transactions ?

It's a good idea always to specify destroy-method if you are using external connection pool provider , so as to shutdown the data source immediately when the spring application shuts down

<bean id="basisDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">

Change the following in your c3p0 pool settings

Take two approaches Remove maxConnectionAge & maxIdleTime

Test by changing below two parameters , they are expensive in terms of performance hit

testConnectionOnCheckout =true 
preferredTestQuery =SELECT 1

In case if you are able to resolve it using the above settings

Change the above settings to below as the below is less expensive in terms of performance hit

idleConnectionTestPeriod – 30 
preferredTestQuery =SELECT 1

Post your findings

Answer 2

You need periodic connection testing so that network appliances like firewalls don't kill the connection. All the details here: C3P0 configuration

Answer 3

I would check

• whether mysql is not closing the connections from its side - what is the value of wait_timeout in your my.cnf?

• if hibernate is using its own internal connection pool , you should have hibernate.connection.provider_class set, can you post your hibernate configuration? see http://docs.jboss.org/hibernate/orm/4.1/devguide/en-US/html_single/#d5e150

Answer 4

I suggest, first adding a destroy method in your application beans configuration as suggested by 'user395072'

Other than that, check the mySQL connection and server configuration properties. I believe, mySQL has some 1000 queries default limit (I saw this on mySQL 5.24). Disconnection is, to my best guess, happening at the mySQL tier - where the server might be 'thinking' that the connection is no longer needed. If there was a fixed time-frame to 'disconnect' or any user event associated with the disconnections then there would have been a possibility of application configuration issues.

In your case, I strongly think, it is the database server configuration.