Azure application gateway returning random 502 error

Question

We have a setup as follow on Azure

The problem we have been encountering is that AG is reporting 502 errors on IP1 randomly. In a day it could get about 20 502 errors on some random files. However, these files on IP1's IIS log also shows success in returning data.

IP2 doesn't have this issue at all.

We've tried to cross check the date/time from AG hitting 502 and comparing to IP1's IIS log to find the corresponding request but could not find any. Assumption here is that if the request reached IP1 it would have created an entry in IIS log.

Because the 502 errors are random, and reported only by AG we are not able to lock down the root cause.

Anyone have encountered such issue before or know how to troubleshoot such issue?

Answer 1

you can try these out :

I understand that the backend Win VM's are running IIS and they are being load balanced by the Application Gateway. Random 502,

1. Check the HTTP timeout value; ask the helpdesk team to check how long the request takes to be processed and sometimes if the processing time is > than set HTTP time out. Random 502 is experienced.

2. Check backend health at the time of the issue; If shows unhealthy - check the corresponding reason

3. Check the failed request count for the Application gateway backend pool during the time of the issue

4. Check usage, maybe the CPU might be shooting upwards of 90% - possible

5. Furthermore, you can check the IIS logs for HTTP error logs to check if the IIS is giving the error code 502.

6. If you have WAF enabled, ask the respective team to check the file size limit for upload and download with WAF on

I strongly think its a timeout on the backend Http settings that's causing this as it's intermittent and IIS shows all 200-399 status codes. Try increasing the timeout and test them.

I hope this helps :)