Spring Boot app can't connect to Google Cloud SQL (PostgreSQL) when deploying to Google App Engine

Question

I have a Gradle based Spring Boot (version 2.1.6) app that I'm deploying to Google App Engine. I've created a Google Cloud SQL (PostgreSQL 11) database instance that I'm trying to connect to.

I've followed all the steps in this example: https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-sql-postgres-sample .

When I run the application locally everything works like it should. But when I'm deploying the app to App Engine it can't connect to the database. These are the error's I'm getting:

ERROR: (gcloud.app.deploy) Error Response: [9] 
Application startup error:
    at com.google.api.client.googleapis.services.json.AbstractGoogleJsonClientRequest.newExceptionOnError(AbstractGoogleJsonClientRequest.java:40) ~[google-api-client-1.28.0.jar!/:1.28.0]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest$1.interceptResponse(AbstractGoogleClientRequest.java:417) ~[google-api-client-1.28.0.jar!/:1.28.0]
    at com.google.api.client.http.HttpRequest.execute(HttpRequest.java:1089) ~[google-http-client-1.30.1.jar!/:na]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:515) ~[google-api-client-1.28.0.jar!/:1.28.0]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.executeUnparsed(AbstractGoogleClientRequest.java:448) ~[google-api-client-1.28.0.jar!/:1.28.0]
    at com.google.api.client.googleapis.services.AbstractGoogleClientRequest.execute(AbstractGoogleClientRequest.java:565) ~[google-api-client-1.28.0.jar!/:1.28.0]
    at com.google.cloud.sql.core.CloudSqlInstance.fetchEphemeralCertificate(CloudSqlInstance.java:327) ~[jdbc-socket-factory-core-1.0.14.jar!/:na]
    ... 10 common frames omitted

2019-10-22 20:49:13.206  INFO 1 --- [           main] o.hibernate.jpa.internal.util.LogHelper  : HHH000204: Processing PersistenceUnitInfo [
    name: default
    ...]
2019-10-22 20:49:13.474  INFO 1 --- [           main] org.hibernate.Version                    : HHH000412: Hibernate Core {5.3.10.Final}
2019-10-22 20:49:13.485  INFO 1 --- [           main] org.hibernate.cfg.Environment            : HHH000206: hibernate.properties not found
2019-10-22 20:49:14.420  INFO 1 --- [           main] o.hibernate.annotations.common.Version   : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
2019-10-22 20:49:15.392 ERROR 1 --- [           main] o.s.b.web.embedded.tomcat.TomcatStarter  : Error starting Tomcat context. Exception: org.springframework.beans.factory.UnsatisfiedDependencyException. Message: Error creating bean with name 'authenticationTokenFilter': Unsatisfied dependency expressed through field 'apiKeysService'; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'apiKeysServiceImpl' defined in URL [jar:file:/app.jar!/BOOT-INF/classes!/nl/storegear/sgapi/services/servicesimpl/ApiKeysServiceImpl.class]: Unsatisfied dependency expressed through constructor parameter 0; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'apiKeyRepository': Cannot create inner bean '(inner bean)#72e34f77' of type [org.springframework.orm.jpa.SharedEntityManagerCreator] while setting bean property 'entityManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name '(inner bean)#72e34f77': Cannot resolve reference to bean 'entityManagerFactory' while setting constructor argument; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]
2019-10-22 20:49:15.466  INFO 1 --- [           main] o.apache.catalina.core.StandardService   : Stopping service [Tomcat]
2019-10-22 20:49:15.473  WARN 1 --- [           main] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
 sun.misc.Unsafe.park(Native Method)
 java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
 java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
 java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
 java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 java.lang.Thread.run(Thread.java:748)
2019-10-22 20:49:15.478  WARN 1 --- [           main] o.a.c.loader.WebappClassLoaderBase       : The web application [ROOT] appears to have started a thread named [pool-1-thread-2] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
...

Here's what my application.properties file looks like:

...

spring.datasource.hikari.connectionTimeout=20000
spring.datasource.hikari.maximumPoolSize=5

spring.datasource.username={REDACTED}
spring.datasource.password={REDACTED}
spring.cloud.gcp.sql.database-name={REDACTED}
spring.cloud.gcp.sql.instance-connection-name={REDACTED}
spring.datasource.continue-on-error=true
spring.datasource.initialization-mode=always
spring.cloud.gcp.credentials.location=file:src/main/resources/{REDACTED}.json
spring.jpa.hibernate.ddl-auto=update
spring.jpa.properties.hibernate.temp.use_jdbc_metadata_defaults=false
spring.jpa.show-sql=false

...

I think it might have something to do with the gcp credentials json file but I don't know what to try next. Any ideas?

Answer 1

There may be 2 issues at play here.

1-

When testing the application locally, it was able to connect and get authenticated since the “application.properties” file was properly configured, with the path to the private key specified in the “..credentials.location” property. When App Engine is deployed, it is not the application that tries to connect, but the App Engine instance itself which is serving the application. What I suspect happens in this case, App Engine is not told where to go get the proper credentials which reside in the “application.properties” Spring Boot file.

In this tutorial , a similar integration is done where a Spring Boot application running on App Engine connects to a Cloud SQL instance. The method used to point App Engine to the credentials is done via Spring Profiles holding the credentials , as well as a configuration of the “ app.yaml ” file where the profiles are made active via the “spring_profiles_active” environment variable.

Alternatively, authentication could be done in the Cloud SDK as explained in step 5 of the example you referenced . Here is the command used to acquire the credentials via this method, and here is an implementation example from a complete tutorial.

2-

Spring Boot starters use Tomcat as the default servlet containers . App Engine uses Jetty as servlet containers . When deploying App Engine, if the Tomcat dependencies are not ignored, they will conflict with Jetty's. This may be what's happening when the “osbweb.embedded.tomcat.TomcatStarter: Error starting Tomcat context” error was thrown when executing the App Engine deployment.

Here is the official Spring documentation on how to exclude those dependencies, and here is an implementation example in a complete tutorial which integrates the same components.