Unexpected latency issues AWS-API Gateway

Question

I need help to troubleshoot AWS API gateway latency issues. We have same configuration and even data everything same but facing high latency issues in Non Prod. Actually we are using Nlb and VPC link for API Gateway. Please find same values here below.

We have copied the data from dev mongo to test environment to make sure the same volume of data is present in both the places. We hit /test/16 from both the environment, but experiencing very high latency in dev as compared to sandbox.

Test: Request:/test/16

Status:200

Latency:213ms

Dev:

Request:/test/16

Status:200

Latency:4896ms

Answer 1

Have you checked your VPC logs to see the flow paths for the requests? If not, I suggest starting there.

As FYI, you can learn about VPC flow logs at https://docs.aws.amazon.com/vpc/latest/userguide/flow-logs.html#working-with-flow-logs .

Answer 2

What is behind the load balancer? Anything you are reaching for with DNS names or just IPs?

We had a similar problem at one point, looking in the monitoring of the load balancer(ELB) we found that the problem was downstreams.

The monitoring even showed that we got 504s in the load balancer.

In our case it was DNS caching that caused it, the target instances had been replaced and the DNS in some nginx instances, on the network path to the target, had not been updated.

The nginx instances had to be updated with dynamic DNS resolving. Since nginx default only resolved the target on startup.

With out knowing your architecture however, hard to say what can cause your problems. Here is another DNS story, with some debugging examples: https://srvaroa.github.io/kubernetes/migration/latency/dns/java/aws/microservices/2019/10/22/kubernetes-added-a-0-to-my-latency.html

Good luck.