Azure Web App Service“容器不健康”错误，强制容器每 15 分钟重新启动一次

Question

I have a dot net core 3.1 app running in an Azure Web App environment (Linux). The app is running fine in local environment/debug. When publishing to Azure, the app runs, but in the Applictions logs, I see the container is restarting every ~15min due to unhealthy container message. The memory and processor load is well below 50% and there is no special load on the application.

I dont know why it shows "unhealthy" in the logs and restarts the container so frequently. Is there anything missing in the configuration settings?

Why does the container restart frequently and where to look for more error specific information to debug this issue?

Any hint is welcome.

Application Logs:

2020-10-06T08:08:57.693Z INFO  - Starting container for site
2020-10-06T08:08:57.693Z INFO  - docker run -d -p 5159:8080 --name wttcycleswebapp_0_8c7e9ba1 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:08:58.845Z INFO  - Initiating warmup request to container wttcycleswebapp_0_8c7e9ba1 for site wttcycleswebapp
2020-10-06T08:09:04.094Z INFO  - Container wttcycleswebapp_0_8c7e9ba1 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:24:34.102Z ERROR - Container for wttcycleswebapp_0_8c7e9ba1 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:24:34.102Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:27:15.247Z INFO  - Starting container for site
2020-10-06T08:27:15.248Z INFO  - docker run -d -p 6236:8080 --name wttcycleswebapp_0_6391dc4a -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:27:16.384Z INFO  - Initiating warmup request to container wttcycleswebapp_0_6391dc4a for site wttcycleswebapp
2020-10-06T08:27:20.657Z INFO  - Container wttcycleswebapp_0_6391dc4a for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:36:35.799Z ERROR - Container for wttcycleswebapp_0_6391dc4a site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:36:35.799Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:36:38.668Z INFO  - Starting container for site
2020-10-06T08:36:38.669Z INFO  - docker run -d -p 3648:8080 --name wttcycleswebapp_0_6ad6dbd7 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:36:39.633Z INFO  - Initiating warmup request to container wttcycleswebapp_0_6ad6dbd7 for site wttcycleswebapp
2020-10-06T08:36:43.884Z INFO  - Container wttcycleswebapp_0_6ad6dbd7 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T08:51:38.900Z ERROR - Container for wttcycleswebapp_0_6ad6dbd7 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T08:51:38.901Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T08:51:39.775Z INFO  - Starting container for site
2020-10-06T08:51:39.776Z INFO  - docker run -d -p 3002:8080 --name wttcycleswebapp_0_f59f6020 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T08:51:40.854Z INFO  - Initiating warmup request to container wttcycleswebapp_0_f59f6020 for site wttcycleswebapp
2020-10-06T08:51:45.135Z INFO  - Container wttcycleswebapp_0_f59f6020 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T09:23:50.153Z ERROR - Container for wttcycleswebapp_0_f59f6020 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T09:23:50.154Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T09:23:50.958Z INFO  - Starting container for site
2020-10-06T09:23:50.959Z INFO  - docker run -d -p 3270:8080 --name wttcycleswebapp_0_0c8ce05e -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T09:24:01.308Z INFO  - Initiating warmup request to container wttcycleswebapp_0_0c8ce05e for site wttcycleswebapp
2020-10-06T09:24:01.600Z INFO  - Container wttcycleswebapp_0_0c8ce05e for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:00:33.648Z INFO  - Starting container for site
2020-10-06T11:00:33.649Z INFO  - docker run -d -p 4757:8080 --name wttcycleswebapp_2_de12fb53 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:00:35.792Z INFO  - Initiating warmup request to container wttcycleswebapp_2_de12fb53 for site wttcycleswebapp
2020-10-06T11:00:42.307Z INFO  - Container wttcycleswebapp_2_de12fb53 for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:16:17.316Z ERROR - Container for wttcycleswebapp_2_de12fb53 site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T11:16:17.316Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T11:16:24.769Z INFO  - Starting container for site
2020-10-06T11:16:24.769Z INFO  - docker run -d -p 1185:8080 --name wttcycleswebapp_0_3904d3bf -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:16:25.662Z INFO  - Initiating warmup request to container wttcycleswebapp_0_3904d3bf for site wttcycleswebapp
2020-10-06T11:16:36.943Z INFO  - Container wttcycleswebapp_0_3904d3bf for site wttcycleswebapp initialized successfully and is ready to serve requests.
2020-10-06T11:31:56.959Z ERROR - Container for wttcycleswebapp_0_3904d3bf site wttcycleswebapp is unhealthy, Stopping site.
2020-10-06T11:31:56.959Z INFO  - Stopping site wttcycleswebapp because it is not healthy.
2020-10-06T11:31:58.658Z INFO  - Starting container for site
2020-10-06T11:31:58.659Z INFO  - docker run -d -p 2576:8080 --name wttcycleswebapp_0_590a51f3 -e WEBSITE_SITE_NAME=WTTCyclesWebApp -e WEBSITE_AUTH_ENABLED=False -e WEBSITE_ROLE_INSTANCE_ID=0 -e WEBSITE_HOSTNAME=wttcycleswebapp.azurewebsites.net -e WEBSITE_INSTANCE_ID=92c475eedd41771f9c6f8d7f8c52ad11b610d1d082a92269eb132b8b216a0d8d -e HTTP_LOGGING_ENABLED=1 appsvc/dotnetcore:3.1-latest_20200717.2 dotnet Cycles.dll 

2020-10-06T11:31:59.684Z INFO  - Initiating warmup request to container wttcycleswebapp_0_590a51f3 for site wttcycleswebapp
2020-10-06T11:32:03.912Z INFO  - Container wttcycleswebapp_0_590a51f3 for site wttcycleswebapp initialized successfully and is ready to serve requests.

In addition, I can see in the "container crash" log:

Container exited unexpectedly: last 10 seconds logs [
2020-10-06T11:16:14.928728537Z Could not open output /home/logs/dumps/coredump.f7d267b6a38a.28.1601982035: No such file or directory
2020-10-06T11:16:14.931442136Z Writing minidump with heap to file /home/logs/dumps/coredump.f7d267b6a38a.28.1601982035
2020-10-06T11:16:15.118711399Z /opt/startup/startup.sh: line 10:    28 Aborted                 (core dumped) dotnet Cycles.dll
]

Answer 1

It looks like your application is crashing when trying to write logs on the file system. This could be the cause of failure if your logs are written to the disk or rotated every 15 minutes. As a good practise, it's probably better to redirect the logs to standard console output instead of writing them to the file system.

Another potential explanation could be a permission issue on the log location

Answer 2

Agree with totor that it is related to the log files.

That it works locally but not in Azure gives a clue to what may be the root cause.

It may be that you need to enable container logs.

App Service -> Monitoring -> App Service logs : Then turn on file system logging

Answer 3

Your problem could be related with service quotas .

Depending of your app service plan, your application can be subject to certain limits on the resources it can use.

Free and shared apps are more constrained, but even with other service plans the filesystem quota always applies.

As the documentation indicates, if the app memory quota is exceeded, the app is stopped temporarily; if the filesystem quota is exceeded, any write operation fails, even writes to logs.

Either of these two reasons or something else related to quotas could be the source of your problem.

You can check the quotas and limits of every app service plan here .

You can monitor your app service quotas metrics from the option App Service Plan > Quotas in your app service detail page on Azure portal.

Answer 4

It seems your App Service has a Health Check configured and it is failing, causing Azure to restart it periodically.

On your App Service's configuration, go to Monitoring > Health check and check if it is enabled. If it is, try disabling it:

Remember to click Save

In my case, Azure still restarted the App Service at least once after I disabled the health checks. To be on the safe side, I recommend that you stop and start the App Service after disabling the health check.

Answer 5

Validate your health check path exists and works. As it says in the image below,

Health check configuration changes will restart your app. To minimize impact to production apps, we recommend setting it up on a staging slot and swapping it into production.

An invalid health check will restart your app.

My problem was an invalid health Path . My application handles /health , but we previously defined Path as /health/ which apparently is not the same.

Note that disabling your health check won't affect slot swapping.

Answer 6

Had this same issue when I deployed an Azure Web App.

I was getting the error:

Your app is unhealthy. Click here for details

Here's how I fixed it :

The issue was that since my Application did not have a defined path for Health Check, I wanted to use the homepage/root path as the Health Check Path.

For this, I was specifying the Health Check Path as the URL of App Service ( api.mywebsite.com ) instead of the root path ( / )

Changing the Health Check Path from the URL of App Service ( api.mywebsite.com ) to the root path ( / ) solved the issue for me.