Erratic Google Cloud Engine - Standard Environment - Latency (Spring Boot App)

Question

Attempting to deploy a springboot app to Google App Engine (standard environment with java11, with the highest spec instance possible). The following is our app.yaml:

  runtime: java11
  entrypoint: java -jar portal-0.0.1-SNAPSHOT.war
  env: standard
  instance_class: B8
  handlers:
    - url: /.*
      script: ignored field
  manual_scaling:
  instances: 1

Functionally the application runs fine.

Technically the page response latency is unpredictable (often terrible)! Sometimes responses take a fraction of a second, sometimes 10 seconds. This is only seen when deploying to google cloud, and not in local testing (all requests take sub 1 second deployed locally). Logs verify that the time executing code within the controller is negligible.

After googling, I found suggestions to run a formatted curl query. Executing this curl query five times in a row yielded the following results (showing the latency appears to be in time_starttransfer):

 - time_namelookup: 0.015000time_connect: 0.031000time_appconnect:
   0.078000time_pretransfer: 0.078000time_redirect: 0.000000time_starttransfer: 3.844000"time_total: 3.906000
 - time_namelookup: 0.015000time_connect: 0.015000time_appconnect:
   0.062000time_pretransfer: 0.062000time_redirect: 0.000000time_starttransfer: 7.469000"time_total: 7.547000
 - time_namelookup: 0.015000time_connect: 0.031000time_appconnect:
   0.109000time_pretransfer: 0.109000time_redirect: 0.000000time_starttransfer: 0.156000"time_total: 0.219000
 - time_namelookup: 0.015000time_connect: 0.015000time_appconnect:
   0.047000time_pretransfer: 0.047000time_redirect: 0.000000time_starttransfer: 3.781000"time_total: 3.843000
 - time_namelookup: 0.000001time_connect: 0.015000time_appconnect:
   0.062000time_pretransfer: 0.062000time_redirect: 0.000000time_starttransfer: 0.094000"time_total: 0.094000

Any ideas on how to test or isolate the underlying issue would be appreciated. Also if anybody has experienced anything like this (or has run any similar configurations without issue) I would be keen to know.

Thanks in advance.

Answer 1

Keep in mind that Stackoverflow is for more precise answers and questions. Since your question is how to isolate the issue based on available tools Google has, it would be interesting to see what happens to App Engine when it receives the request. For example, trace logs of the request. To know the breakdown of the execution when the request is received.

Also, what kind of requests does this... For example, would caching/static files have an impact or is it all types of requests. What happens with another war file? etc...

The general idea would be to identify what takes that long and you do that by looking at traces or files slowing down the execution (assuming the issue isn't app engine, although your config of manual scaling looks good).