Why is my website experiencing random slow api requests?

Question

I have a VB.NET/Vue website hosted on an internal IIS 8.5 Windows 2012R2 Server. Our company has about 30 users using the site at any given time. The users are experiencing random delays throughout the day and on some days there's no delays (site works great most of the time). What I'm looking for is any suggestions on where to start looking to solve the issue. Here's what I've found so far.

1. User goes to site and initiates an api request from the UI
2. User sees a loading icon for anywhere up to a minute or so while the request returns
3. The request eventually reaches the server after some time and executes really fast within milliseconds and returns the response to the user
4. By this time, many users have already refreshed the page making new requests that succeed on page load. For the users that are patient and wait for the response, it eventually returns the response.

Here's some screenshots:

So to sum everything up, there are several users experiencing delays on a daily basis.

Some days we don't have any delays, but most days we have several users experiencing multiple delays of several seconds to 30 seconds to 1 minute.

I've found all this using LogRocket and NewRelic and what is happening is all these requests are completing within milliseconds, but the request doesn't seem to reach the server for some period of time.

I've been monitoring the CPU/Memory/Network on these servers and it all seems fine to me during when these issues occur.

It seems that the problem lies between the users computer and whatever hardware/software exists before reaching the web server.

Update here... Found that the problem is occurring on the users computer in all these instances. Using google chrome's performance api, I was able to track timing info for these requests and found that the problem is in the fetchStart. So whatever is happening here is the cause of the issue.

Example below:

entryType: resource startTime: 1119531.820000033 duration: 56882.43999995757 initiatorType: xmlhttprequest nextHopProtocol: http/1.1 workerStart: 0 redirectStart: 0 redirectEnd: 0 fetchStart: 1119531.820000033 domainLookupStart: 1176401.0199999902 domainLookupEnd: 1176402.2699999623 connectStart: 1176402.2699999623 connectEnd: 1176404.8350000521 secureConnectionStart: 1176403.6700000288 requestStart: 1176404.8549999716 responseStart: 1176413.5300000198 responseEnd: 1176414.2599999905 transferSize: 15145 encodedBodySize: 14884 decodedBodySize: 14884 serverTiming: [] workerTiming: []

fetchStart is at 1119531.820000033, then requestStart is at 1176404.8549999716 so the problem is something between fetchStart and requestStart. Still looking into what is causing this.

Answer 1

In 20202, we are experiencing something very similar with a small fraction of our customers. There is a significant gap between the timing api requestStart and the startTime . This gap can be up to 8 minutes -- I admire the patience of customers waiting that long. The wait periods are also close to multiples of a minute.

In our case, it appears that there is a (transparent?) proxy between those browsers and our server infrastructure which appears to be triggering the problem. In particular, it forces a downgrade of HTTP/2 to HTTP/1.1. Whitelisting our website in that proxy does solve the problem. This isn't a very satisfactory solution, but it does make the customer happier!