Spring Boot Rest Service raising JPA connection error after hours of working correctly

Question

I have a Spring Boot Rest service deployed as war on a Tomcat 7 container on a linux ubuntu server.

The rest service just expose a /detections endpoint which returns a JSON representation of the last N records of a table in a mysql database (which runs on the same server). The database is accessed by my controller via Spring Data JPA.

After deploying the service to tomcat, the service seems to work perfectly, and this lasts for several hours. If I check though, say, the morning after, the service seems to have crashed due to a JPA exception (see below). If I sudo service tomcat7 restart, the service begins again to work correctly, and this again only lasts a few hours.

Here is the logback output, where I tried to isolate the most relevant lines

*(webapp working correctly for a number of hours)*
.
.
.
155.    2016-05-04 14:07:00,553 INFO [http-bio-8080-exec-26] o.h.t.h.SchemaUpdate [SchemaUpdate.java:218] HHH000102: Fetching database metadata
156.    2016-05-04 14:07:04,581 ERROR [http-bio-8080-exec-26] o.h.t.h.SchemaUpdate [SchemaUpdate.java:226] HHH000319: Could not get database metadata
157.    com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
.
.
381.    Caused by: java.sql.SQLException: Access denied for user 'myuser'@'localhost' (using password: YES)
.
.
.
391.    2016-05-04 14:07:04,588 ERROR [http-bio-8080-exec-26] o.h.t.h.SchemaUpdate [SchemaUpdate.java:272] HHH000299: Could not complete schema update
392.    com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
.
.
.
615.    2016-05-04 14:07:06,854 INFO [http-bio-8080-exec-26] o.s.b.a.s.AuthenticationManagerConfiguration [AuthenticationManagerConfiguration.java:170] 
616.
617.    Using default security password: 161b39f3-9d55-453f-b9e9-310244de426b
.
.
.
851.    2016-05-04 14:07:24,649 WARN [http-bio-8080-exec-26] o.h.j.i.EntityManagerFactoryRegistry [EntityManagerFactoryRegistry.java:80] HHH000436: Entity manager factory name (default) is already registered.  If entity manager will be clustered or passivated, specify a unique value for property 'hibernate.ejb.entitymanager_factory_name'
.
.
.
902.    2016-05-04 14:37:55,360 INFO [http-bio-8080-exec-28] o.s.w.s.DispatcherServlet [FrameworkServlet.java:485] FrameworkServlet 'dispatcherServlet': initialization started
903.    2016-05-04 14:37:55,377 INFO [http-bio-8080-exec-28] o.s.w.s.DispatcherServlet [FrameworkServlet.java:504] FrameworkServlet 'dispatcherServlet': initialization completed in 17 ms
904.    2016-05-04 14:37:59,849 WARN [http-bio-8080-exec-28] o.h.e.j.s.SqlExceptionHelper [SqlExceptionHelper.java:144] SQL Error: 0, SQLState: 08001
905.    2016-05-04 14:37:59,849 ERROR [http-bio-8080-exec-28] o.h.e.j.s.SqlExceptionHelper [SqlExceptionHelper.java:146] Could not create connection to database server. Attempted reconnect 3 times. Giving up.
906.    2016-05-04 14:38:00,297 ERROR [http-bio-8080-exec-28] o.s.b.c.w.ErrorPageFilter [ErrorPageFilter.java:177] Forwarding to error page from request [/detections] due to exception [Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection]
907.    org.springframework.transaction.CannotCreateTransactionException: Could not open JPA EntityManager for transaction; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.JDBCConnectionException: Could not open connection
.
.
.
*(similar error patterns repeat)*

This is my application.properties file (of course, mydatabase, myuser and mypassword replace actual connection data) :

spring.jpa.database-platform=org.hibernate.dialect.MySQL5InnoDBDialect
spring.datasource.driverClassName=com.mysql.jdbc.Driver
spring.datasource.url=jdbc:mysql://localhost:3306/mydatabase?autoReconnect=true
spring.datasource.username=myuser
spring.datasource.password=mypassoword
spring.jpa.hibernate.ddl-auto=update
spring.data.jpa.repositories.enabled=true
spring.jpa.show-sql=true

In mysql, I granted all access to mydatabase to user 'myuser'@'localhost' and actually I can state that the problem is not related to wrong credentials since, as I mentioned the service actually manage to connect for a good number of hours. The ?autoReconnect=true was added in a second stage trying to solve the problem, but nothing changed.

Here is the (pretty simple) code of my REST controller (again, only the relevant parts).

@RestController
@Transactional
public class DetectionController {

    @Autowired
    public DetectionRepository detectionRepository;

    .
    .
    .

    @RequestMapping(value="/detections", method=RequestMethod.GET  )
    public Object getDetectionsLimit( @RequestParam(required=false, defaultValue="0") int limit )
    {
        try {
            List<Detection> detections = detectionRepository.findTop5ByOrderByTimestampDesc(); 
            return detections;
        } catch (final Exception e ) {
            return new Object() {
                public String error = e.getMessage();
            };
        }
    }

    .
    .
    .

}

Thanks in advance for any help

Answer 1

I had a similar problem on mine and adding the properties mentioned below fixed my issue you can give it a try.

spring.datasource.test-on-borrow= true
spring.datasource.validation-query= SELECT 1