Hibernate sometimes takes far too long to obtain JDBC connection

Question

We're using Hibernate quite successfully for the most part in our Mule application. However occasionally we are finding it's taking minutes to write from the application to the database and we need to find out why.

Here are the Spring and Hibernate logs from one such occasion.

[2018-10-18 05:20:42.156] DEBUG   org.springframework.security.authentication.ProviderManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Authentication attempt using org.springframework.security.authentication.dao.DaoAuthenticationProvider
[2018-10-18 05:20:42.160] DEBUG   org.springframework.beans.CachedIntrospectionResults [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Not strongly caching class [com.company.mule.core.component.SynchronouslySaveOrUpdateImportStatusDtoComponent] because it is not cache-safe
[2018-10-18 05:20:42.160] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Creating new transaction with name [com.company.service.ImportStatusService.saveOrMerge]: PROPAGATION_REQUIRES_NEW,ISOLATION_DEFAULT,timeout_15; ''
[2018-10-18 05:20:42.160] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Opened new EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@50302b40] for JPA transaction
[2018-10-18 05:20:42.160] DEBUG   org.hibernate.engine.transaction.spi.AbstractTransactionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: begin
[2018-10-18 05:20:42.160] DEBUG   org.hibernate.engine.jdbc.internal.LogicalConnectionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Obtaining JDBC connection
.
.
.
[2018-10-18 05:54:07.470] DEBUG   org.hibernate.engine.jdbc.internal.LogicalConnectionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Obtained JDBC connection
[2018-10-18 05:54:07.470] DEBUG   org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: initial autocommit status: true
[2018-10-18 05:54:07.470] DEBUG   org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: disabling autocommit
[2018-10-18 05:54:07.567] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Exposing JPA transaction as JDBC transaction [org.springframework.orm.jpa.vendor.HibernateJpaDialect$HibernateConnectionHandle@2bbdca69]
[2018-10-18 05:54:07.568] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Found thread-bound EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@50302b40] for JPA transaction
[2018-10-18 05:54:07.568] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Participating in existing transaction
[2018-10-18 05:54:07.568] DEBUG   org.hibernate.SQL [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: select this_.ID as ID1_6_1_, this_.CREATED_DATE as CREATED_2_6_1_, this_.MODIFIED_DATE as MODIFIED3_6_1_, this_.UUID as UUID4_6_1_, this_.VERSION as VERSION5_6_1_, this_.import_frequency as import_f6_6_1_, this_.import_type_id as import_t8_6_1_, this_.TENANT_ID as TENANT_I7_6_1_, type1_.ID as ID1_5_0_, type1_.CREATED_DATE as CREATED_2_5_0_, type1_.MODIFIED_DATE as MODIFIED3_5_0_, type1_.UUID as UUID4_5_0_, type1_.VERSION as VERSION5_5_0_, type1_.CODE as CODE6_5_0_, type1_.NAME as NAME7_5_0_ from import_type_detail this_ inner join import_type type1_ on this_.import_type_id=type1_.ID where type1_.CODE=? and this_.TENANT_ID=?
[2018-10-18 05:54:07.568] INFO    std-out [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Hibernate: select this_.ID as ID1_6_1_, this_.CREATED_DATE as CREATED_2_6_1_, this_.MODIFIED_DATE as MODIFIED3_6_1_, this_.UUID as UUID4_6_1_, this_.VERSION as VERSION5_6_1_, this_.import_frequency as import_f6_6_1_, this_.import_type_id as import_t8_6_1_, this_.TENANT_ID as TENANT_I7_6_1_, type1_.ID as ID1_5_0_, type1_.CREATED_DATE as CREATED_2_5_0_, type1_.MODIFIED_DATE as MODIFIED3_5_0_, type1_.UUID as UUID4_5_0_, type1_.VERSION as VERSION5_5_0_, type1_.CODE as CODE6_5_0_, type1_.NAME as NAME7_5_0_ from import_type_detail this_ inner join import_type type1_ on this_.import_type_id=type1_.ID where type1_.CODE=? and this_.TENANT_ID=?
[2018-10-18 05:54:07.671] DEBUG   org.hibernate.loader.Loader [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Result row: EntityKey[com.company.domain.ImportType#53], EntityKey[com.company.domain.ImportTypeDetail#71]
[2018-10-18 05:54:07.671] DEBUG   org.hibernate.engine.internal.TwoPhaseLoad [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Resolving associations for [com.company.domain.ImportType#53]
[2018-10-18 05:54:07.671] DEBUG   org.hibernate.engine.internal.TwoPhaseLoad [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Done materializing entity [com.company.domain.ImportType#53]
[2018-10-18 05:54:07.671] DEBUG   org.hibernate.engine.internal.TwoPhaseLoad [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Resolving associations for [com.company.domain.ImportTypeDetail#71]
[2018-10-18 05:54:07.671] DEBUG   org.hibernate.engine.internal.TwoPhaseLoad [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Done materializing entity [com.company.domain.ImportTypeDetail#71]
[2018-10-18 05:54:07.768] DEBUG   org.hibernate.stat.internal.ConcurrentStatisticsImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: HHH000117: HQL: [CRITERIA] select this_.ID as ID1_6_1_, this_.CREATED_DATE as CREATED_2_6_1_, this_.MODIFIED_DATE as MODIFIED3_6_1_, this_.UUID as UUID4_6_1_, this_.VERSION as VERSION5_6_1_, this_.import_frequency as import_f6_6_1_, this_.import_type_id as import_t8_6_1_, this_.TENANT_ID as TENANT_I7_6_1_, type1_.ID as ID1_5_0_, type1_.CREATED_DATE as CREATED_2_5_0_, type1_.MODIFIED_DATE as MODIFIED3_5_0_, type1_.UUID as UUID4_5_0_, type1_.VERSION as VERSION5_5_0_, type1_.CODE as CODE6_5_0_, type1_.NAME as NAME7_5_0_ from import_type_detail this_ inner join import_type type1_ on this_.import_type_id=type1_.ID where type1_.CODE=? and this_.TENANT_ID=?, time: 200ms, rows: 1
[2018-10-18 05:54:07.768] DEBUG   org.hibernate.engine.spi.ActionQueue [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Executing identity-insert immediately
[2018-10-18 05:54:07.768] DEBUG   org.hibernate.SQL [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: insert into import_status (CREATED_DATE, MODIFIED_DATE, UUID, VERSION, ASSOCIATED_DATA, ELAPSED_TIME, ERROR_MODEL_COUNT, HASH, IMPORT_DATE, IMPORT_NAME, IMPORT_PROGRESS, IMPORT_TYPE_DETAILS_ID, PROCESSED_MODEL_COUNT, TENANT_ID, TOTAL_MODEL_COUNT) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[2018-10-18 05:54:07.769] INFO    std-out [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Hibernate: insert into import_status (CREATED_DATE, MODIFIED_DATE, UUID, VERSION, ASSOCIATED_DATA, ELAPSED_TIME, ERROR_MODEL_COUNT, HASH, IMPORT_DATE, IMPORT_NAME, IMPORT_PROGRESS, IMPORT_TYPE_DETAILS_ID, PROCESSED_MODEL_COUNT, TENANT_ID, TOTAL_MODEL_COUNT) values (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
[2018-10-18 05:54:07.867] DEBUG   org.hibernate.id.IdentifierGeneratorHelper [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Natively generated identity: 8957
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.cache.ehcache.internal.regions.EhcacheGeneralDataRegion [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: key: import_status value: 6307193274236928
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.event.internal.AbstractFlushingEventListener [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Processing flush-time cascades
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.event.internal.AbstractFlushingEventListener [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Dirty checking collections
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.engine.internal.Collections [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Collection found: [com.company.domain.ImportStatus.importErrors#8957], was: [<unreferenced>] (initialized)
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.event.internal.AbstractFlushingEventListener [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Flushed: 0 insertions, 0 updates, 0 deletions to 3 objects
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.event.internal.AbstractFlushingEventListener [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Flushed: 1 (re)creations, 0 updates, 0 removals to 1 collections
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.internal.util.EntityPrinter [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Listing entities:
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.internal.util.EntityPrinter [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: com.company.domain.ImportType{createdDate=2018-03-15 14:05:00.0, code=DISPATCH_REPORT, modifiedDate=null, name=Dispatch Report, id=53, uuid=24983801440501787, version=0}
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.internal.util.EntityPrinter [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: com.company.domain.ImportStatus{associatedData=null, errorModelCount=0, uuid=8A897AAE6682E06E016685BCFE5022B5, version=0, importErrors=[], createdDate=Thu Oct 18 05:54:07 UTC 2018, associatedDataLatestId=null, totalModelCount=0, importDate=2018-10-18 05:20:42.009, modifiedDate=null, tenantId=202, importName=DISPATCH_REPORT, id=8957, associatedDataSourceSystem=null, importProgress=NOT_STARTED, processedModelCount=0, hash=null, importTypeDetail=com.company.domain.ImportTypeDetail#71, elapsedTime=0.0}
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.internal.util.EntityPrinter [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: com.company.domain.ImportTypeDetail{importType=com.company.domain.ImportType#53, createdDate=2018-03-15 14:05:00.0, modifiedDate=null, tenantId=202, id=71, importFrequency=DAILY, uuid=24983801440501788, version=0}
[2018-10-18 05:54:07.968] DEBUG   org.hibernate.cache.ehcache.internal.regions.EhcacheGeneralDataRegion [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: key: import_error value: 6307193274236929
[2018-10-18 05:54:07.968] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Initiating transaction commit
[2018-10-18 05:54:07.969] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Committing JPA transaction on EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@50302b40]
[2018-10-18 05:54:07.969] DEBUG   org.hibernate.engine.transaction.spi.AbstractTransactionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: committing
[2018-10-18 05:54:08.068] DEBUG   org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: committed JDBC Connection
[2018-10-18 05:54:08.068] DEBUG   org.hibernate.engine.transaction.internal.jdbc.JdbcTransaction [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: re-enabling autocommit
[2018-10-18 05:54:08.165] DEBUG   org.hibernate.cache.ehcache.internal.regions.EhcacheGeneralDataRegion [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: key: import_status value: 6307193029283840
[2018-10-18 05:54:08.165] DEBUG   org.hibernate.cache.ehcache.internal.regions.EhcacheGeneralDataRegion [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: key: import_error value: 6307193029283840
[2018-10-18 05:54:08.165] DEBUG   org.springframework.orm.jpa.JpaTransactionManager [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Closing JPA EntityManager [org.hibernate.jpa.internal.EntityManagerImpl@50302b40] after transaction
[2018-10-18 05:54:08.165] DEBUG   org.springframework.orm.jpa.EntityManagerFactoryUtils [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Closing JPA EntityManager
[2018-10-18 05:54:08.165] INFO    org.hibernate.engine.internal.StatisticalLoggingSessionEventListener [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Session Metrics {
    2005309943402 nanoseconds spent acquiring 1 JDBC connections;
    0 nanoseconds spent releasing 0 JDBC connections;
    385901 nanoseconds spent preparing 2 JDBC statements;
    199318868 nanoseconds spent executing 2 JDBC statements;
    0 nanoseconds spent executing 0 JDBC batches;
    725792 nanoseconds spent performing 4 L2C puts;
    0 nanoseconds spent performing 0 L2C hits;
    0 nanoseconds spent performing 0 L2C misses;
    417472 nanoseconds spent executing 1 flushes (flushing a total of 3 entities and 1 collections);
    2444 nanoseconds spent executing 1 partial-flushes (flushing a total of 0 entities and 0 collections)
[2018-10-18 05:54:08.165] DEBUG   org.hibernate.engine.jdbc.internal.LogicalConnectionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Releasing JDBC connection
[2018-10-18 05:54:08.166] DEBUG   org.hibernate.engine.jdbc.internal.LogicalConnectionImpl [[prd-globe-import-acmeap].HTTP_Message_Listener_Configuration.worker.57]: Released JDBC connection

Any thoughts on what the problem may be?

Would it be possible at the very least to get it to time out after a reasonable amount of time?

Many thanks, Pete

Answer 1

I would check the database logs to have additional informations. Maybe the database is in lock state when you try to write and you need to wait long.