Debugging ENOTFOUND error using aws sdk v3

Question

I've been using AWS JS SDK V3 and have noticed that my lambdas are intermittently hitting errors connecting to AWS resources. Below I have an example for dynamodb, but I have also had issues connecting to secrets manager. My lambdas and resources are all contained within a VPC. I've noticed that these issues seem to be hit more often during a lambda cold start, but I'm not entirely sure. If a request is resent (user on the frontend refreshes the page) this error seems to go away. I was hoping that the built in client retries would reduce the errors that my code sees, but it appears that no retries are attempted.

I am looking for potential debugging tips that might reveal what is the cause of these issues. So far I've been looking through cloudwatch logs which does not appear to have any good insights. I believe this is being cause by bad DNS resolution, but I am surprised by the frequency of these errors. Stopping short of moving my lambdas to ec2 and utilizing a cache, what are ways in which I can fix this.

Reading this article: https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-cause-of-failed-dns-queries/ suggests increasing the DNS retry timer, but I'm unsure how I would do that as well.

{
    "errorType": "Error",
    "errorMessage": "getaddrinfo ENOTFOUND dynamodb.us-east-1.amazonaws.com",
    "code": "ENOTFOUND",
    "errno": -3008,
    "syscall": "getaddrinfo",
    "hostname": "dynamodb.us-east-1.amazonaws.com",
    "$metadata": {
        "attempts": 1,
        "totalRetryDelay": 0
    },
    "stack": [
        "Error: getaddrinfo ENOTFOUND dynamodb.us-east-1.amazonaws.com",
        "    at GetAddrInfoReqWrap.onlookup [as oncomplete] (dns.js:71:26)"
    ]
}

Answer 1

How do you connect to DynamoDB from within your VPC?

Are you using a NAT instance or gatway?

Are you using custom DNS resolution?

I would suggest that you add a DynamoDB VPCe to your VPC to allow you to connect to DynamoDB via AWS private.network.