Hibernate loses connection

Question

I am using hibernate to connect my mysql database and perform transactions.

I am using a single SessionFactory throughout the application and i don't have other connections to the database, yet, i am receiving the exception below:

java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3466)
    ... 21 common frames omitted
Wrapped by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure

The last packet successfully received from the server was 526 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:425)
    at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3556)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3897)
    at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2524)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2677)
    at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2545)
    at com.mysql.jdbc.ConnectionImpl.setAutoCommit(ConnectionImpl.java:4842)
    at org.hibernate.engine.jdbc.connections.internal.PooledConnections.poll(PooledConnections.java:84)
    at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.getConnection(DriverManagerConnectionProviderImpl.java:186)
    at org.hibernate.internal.NonContextualJdbcConnectionAccess.obtainConnection(NonContextualJdbcConnectionAccess.java:35)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:99)
    ... 11 common frames omitted
Wrapped by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
    at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)
    at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:111)
    at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:97)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.acquireConnectionIfNeeded(LogicalConnectionManagedImpl.java:102)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getPhysicalConnection(LogicalConnectionManagedImpl.java:129)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.getConnectionForTransactionManagement(LogicalConnectionManagedImpl.java:247)
    at org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:254)
    at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:203)
    at org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:56)
    at org.hibernate.internal.AbstractSharedSessionContract.beginTransaction(AbstractSharedSessionContract.java:387)
    at com.kitaplist.common.book.dao.HibernateBookDao.find(HibernateBookDao.java:56)
    at com.kitaplist.common.Collector.lambda$collectMetaBooksAndNewBooks$1(Collector.java:137)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

This is how i create my SessionFactory:

public static SessionFactory getSessionFactory() {

    if (sessionFactory == null) {
        sessionFactory = new Configuration()
                .configure()
                .addAnnotatedClass(Seller.class)
                .addAnnotatedClass(Book.class)
                .buildSessionFactory();
    }

    return sessionFactory;
}

and here is the function that I use in my BookDao:

@Override
public void save(Book book) {
    Session session = sessionFactory.openSession();
    Transaction tx = session.beginTransaction();
    try {
        session.save(book);
    } catch (Exception e) {
        e.printStackTrace();
    } finally {
        try {
            tx.commit();
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            session.close();
        }

    }
}

my application is a crawler crawls a book object from web and saves the object to the database through the above save function. I couldn't find the reason behind this exception.

on the command console, i can see that the connection is re-established after it is lost, here :

SLF4J: A number (289) of logging calls during the initialization phase have been intercepted and are
SLF4J: now being replayed. These are subject to the filtering rules of the underlying logging system.
SLF4J: See also http://www.slf4j.org/codes.html#replay
Wed Mar 14 16:36:29 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Wed Mar 14 16:36:29 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Wed Mar 14 16:36:30 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.
Wed Mar 14 16:36:29 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.


Wed Mar 14 16:47:14 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

Wed Mar 14 16:47:17 UTC 2018 WARN: Establishing SSL connection without server's identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn't set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to 'false'. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

I would appreciate any help.

Answer 1

Since your problem involves network the best idea might be to add a JDBC connection pool to your setup. This will ensure that connections don't get stale while your application is running eg due to database server-side timeout.

HikariCP is by far my favourite pool as it sets sensible defaults eg on-borrow connection validation that can't be disabled. Since you are using Hibernate, HikariCP shows how to set everything up on their wiki page . The right setup however might depend on other libraries you are using eg shared java.sql.DataSource bean in Spring Boot.

There are other pooling libraries you can use but please take few minutes to read below before going that route:

• "My benchmark doesn't show a difference."
• Pool Analysis

If using a pooling library with on-borrow connection validation won't solve this problem you'd have to dive into the network setup. Perhaps between your application and the database there is a firewall which terminates any network connection eg by imposing a hard timeout. Normally this situation is solvable by setting a connection validation interval in the pool which is lower than the timeout imposed by the network.

Answer 2

Based in your problem description, I believe the connection it is dropped because it becomes iddle...

Try to append '?autoReconnect=true' to the end of your database's JDBC URL... and see if the problem does not happen more...

However, if you are not able to connect to the database even once, I suggest to check the following items:

1. Do a ping command to your database IP from the server host
2. Do a telnet command to your database to see if you can reach the database port
3. See if MySql does not have rules about which IPs can talk to it (I known that postgres have this feature, do not know if MySql does)
4. Check that MySql does not have something like drop iddle connections...
5. Check your JDBC connection params

Answer 3

The main culprit is wait_timeout . Its default value is 28800 sec ie 8 hours. From the doc:

The number of seconds the server waits for activity on a noninteractive connection before closing it.

The error you are receiving is caused when the DB connection is idle(not performing any DB query) for wait_timeout secs. After this time, MySQL drops the connection and you when your code makes any DB calls, it gets this error.

Increase this value(to say 1 day) and you can circumvent this problem.

---

However, to fix this issue, put autoReconnect=true in the DB connect string, like below:

jdbc:mysql://db_user:db_user@localhost/mydb?autoReconnect=true

This will cause the code to automatically reconnect the connection when the connection is dropped after wait_timeout secs.

Answer 4

Well when we see log attached it says

The last packet successfully received from the server was 526 milliseconds ago.  The last packet sent successfully to the server was 1 milliseconds ago.

Its not timeout issue for sure. As idle wait time is around 500ms.

java.io.EOFException: Can not read response from server. Expected to read 4 bytes, read 0 bytes before connection was unexpectedly lost.
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3008)


Wrapped by: org.hibernate.exception.JDBCConnectionException: Unable to acquire JDBC Connection
    at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:115)

It looks like Database connection is getting blocked/interrupted.

It can also most likely means that the database has restarted or the network connection to the database has been broken (eg a NAT connection has timed out) or may be firewall issue.

AutoReconnect is not recommended. From MySQL here

Should the driver try to re-establish stale and/or dead connections? If enabled the driver will throw an exception for a queries issued on a stale or dead connection, which belong to the current transaction, but will attempt reconnect before the next query issued on the connection in a new transaction. The use of this feature is not recommended, because it has side effects related to session state and data consistency when applications don't handle SQLExceptions properly, and is only designed to be used when you are unable to configure your application to handle SQLExceptions resulting from dead and stale connections properly. Alternatively, as a last option, investigate setting the MySQL server variable "wait_timeout" to a high value, rather than the default of 8 hours.

Additional Suggestion:

to get rid of this warning : Establishing SSL connection without server's identity verification is

use useSSL=false in Mysql Configuration

eg:

jdbc:mysql://localhost:3306/Peoples?autoReconnect=true&useSSL=false

Answer 5

You can use

sessionFactory.isClosed();

to determine if the connection is still open. Replace your getSessionFactory() method like this.

public static SessionFactory getSessionFactory() {

    if (sessionFactory == null || sessionFactory.isClosed()) {
        sessionFactory = new Configuration()
                .configure()
                .addAnnotatedClass(Seller.class)
                .addAnnotatedClass(Book.class)
                .buildSessionFactory();
    }

    return sessionFactory;
}

Answer 6

   at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:989)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3556)
    at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456)
    at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3897)

From your stacktrace it seems hibernate is not able to obtain connections from database as db server has gone away

    // Check return value, if we get a java.io.EOFException, the server has gone away. We'll pass it on up the exception chain and let someone higher up
   // decide what to do (barf, reconnect, etc).

Check source here:
http://grepcode.com/file/repo1.maven.org/maven2/mysql/mysql-connector-java/5.1.36/com/mysql/jdbc/MysqlIO.java#3774

Possible explaination :
The problem seems at database server side not in your code. You may need to tweak your mysql server settings and problem will be solved. The data you are sending is larger than the packet that is sent over the network to the database.

Causes and solutions :
https://dev.mysql.com/doc/refman/5.7/en/gone-away.html
http://befused.com/mysql/server-has-gone-away

Other Solutions:
in my.cnf file of mysql add following settings

[mysqld]
max_allowed_packet=256M