Google Cloud Run 中的 PHP 服务突然停止响应所有传入请求

Question

I am running a PHP (8.1) backend application in Google Cloud Run. The backend is connected to a MYSQL database running in Google Cloud SQL. Over the last two weeks, we have had three complete outages. The backend server does not respond to any requests, resulting in our app and website being completely down.

Before this happened the first time, the server has been running for many months without any similar problems. I first suspected that this had something to do with some specs on either the backend or the database, but looking at the graphs, I cannot see any obvious reasons why it should all go down.

Notice how the traffic goes down every night, but is still spiky, before it went completely flat around 4PM today:

I then looked at the stats from our Cloud Run server to find any indication there. We are running with relatively high specs, and a flexible container instance size, so this should not cause any troubles like this. Container memory and CPU utilization drops dead all of a sudden. It seems there are no unusual activity going on before the service decided to die.

Our Sentry dashboard shows that there are no captured events from the down time period. However, looking at the logs of the backend service in Google Cloud Logs Explorer, it seems there are heaps of 200 responses in this time interval. By looking at the logs, I don't see any indication that anything is wrong.

The only thing I could think of to resolve this problem, was to redeploy the service inside Google Cloud Run, effectively spinning up a new container with the exact same code and specs. Then it started working again, and has been working since, but I have no idea what have happened. As far as I can think of, we don't have any code or config related changes that could lead to any kind of problems like this.

Does anyone have any thoughts? The only thing I can think of is some sort of memory leak that suddenly gets out of hand. But I assume that should have been able to trace back in some way. If this was the case, I'm also thinking it should have happened more often over a long period of time. Not running nicely for a long time, then down 3 times in 2 weeks.

Any help or pointers would be greatly appreciated!

Answer 1

This error might be caused by expected internal behavior within the networking infrastructure of Cloud Run. Occasionally, maintenance does occur on this networking infrastructure, which can cause some connections to be closed. Since Cloud Run's health checks are not network dependent, this can cause the container to be stuck and unable to communicate with the server. Some Cloud Run applications are sensitive to this network connection drop as well. As you were able to redeploy the Cloud Run container, you were able to "unstick" the container and it was able to run as intended.

A recommended workaround and practice is to implement retry logic within your service code. This will allow the Cloud Run service to be able to reconnect to the servers when a network connection drops.