Why I am getting `Endpoint request timed out` error when AWS SecretsManager is implemented?

Question

I am developing a REST API with AWS Lambda, API Gateway, RDS (MySQL). I am using Node.js.

This is my normal AWS Lambda code, calling database table to get the data and send it over the REST API.

const mysql = require('mysql');
const con = mysql.createConnection({
  host     : "****.****.****.rds.amazonaws.com",
  user     : "****",
  password : "****",
  port     : 3306,
  database : "****"
});

exports.getAllRoles = (event, context, callback) => {
  // allows for using callbacks as finish/error-handlers
  context.callbackWaitsForEmptyEventLoop = false;
  const sql = "select * from role";
  con.query(sql, function (err, result) {
    if (err) throw err;

    var response = {
        "statusCode": 200,
        "headers": {
            "Content-Type": "application/json"
        },
        "body": JSON.stringify(result),
        "isBase64Encoded": false
    };
    callback(null, response)
  });
};

This is the same code, but now with AWS SecretsManager

const mysql = require('mysql');


// Load the AWS SDK
var AWS = require('aws-sdk'),
    region = "us-east-1",
    secretName = "test-secret",
    secret,
    decodedBinarySecret;

// Create a Secrets Manager client
var client = new AWS.SecretsManager({
    region: region
});


exports.getAllRoles = (event, context, callback) => {

    client.getSecretValue({
        SecretId: secretName
    }, function (err, data) {
        if (err) {
            throw err;
        } else {
            // Decrypts secret using the associated KMS CMK.
            // Depending on whether the secret is a string or binary, one of these fields will be populated.
            if ('SecretString' in data) {
                secret = data.SecretString;
            } else {
                let buff = new Buffer(data.SecretBinary, 'base64');
                decodedBinarySecret = buff.toString('ascii');
            }
        }
        // Your code goes here. 
        const secretObj = JSON.parse(secret);

        //Create MySQL Connection
        const con = mysql.createConnection({
            host     : secretObj.host,
            user     : secretObj.user,
            password : secretObj.password,
            port     : secretObj.port,
            database : secretObj.database
          });

        // allows for using callbacks as finish/error-handlers
        context.callbackWaitsForEmptyEventLoop = false;
        const sql = "select * from role";
        con.query(sql, function (err, result) {
            if (err) throw err;

            var response = {
                "statusCode": 200,
                "headers": {
                    "Content-Type": "application/json"
                },
                "body": JSON.stringify(result),
                "isBase64Encoded": false
            };
            callback(null, response)
        });
    });    

};

This is my cloud formation file

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
  aaaa-restapi

  Sample SAM Template for aaaa-restapi
  
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
  Function:
    Timeout: 100
    VpcConfig:
        SecurityGroupIds:
          - sg-4424242424
        SubnetIds:
          - subnet-424242424242

Resources:
  
  GetAllRolesFunction:
    Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
    Properties:
      CodeUri: aaaa-restapi/
      Handler: role-getall.getAllRoles
      Runtime: nodejs14.x
      Events:
        HelloWorld:
          Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
          Properties:
            Path: /role/getall
            Method: get
  

  LambdaRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - lambda.amazonaws.com
            Action:
              - 'sts:AssumeRole'
      Path: /
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyName: root
          PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Effect: Allow
                Action:
                  - ec2:DescribeNetworkInterfaces
                  - ec2:CreateNetworkInterface
                  - ec2:DeleteNetworkInterface
                  - ec2:DescribeInstances
                  - ec2:AttachNetworkInterface
                Resource: '*'

My code without the AWS SecretsManager is working fine when deployed.

However the code with AWS SecretsManager is giving timeout error when deployed and tested the API Call with POSTMAN . I just call

https://*****.****-api.****-1.amazonaws.com/Prod/role/getall

I just see "message": "Endpoint request timed out" and nothing else, even in the console. The same code is working fine when executed with sam invoke local GetAllRolesFunction . Also fine when tested with sam local start-api . The issue only arises when uploaded to AWS and the API Call is made.

What is happening here?

Answer 1

The error message "Endpoint request timed out" from API Gateway suggests that your API-invoked Lambda function took longer than 29 seconds to complete.

I am going to guess that your Lambda function is running in a VPC and you have correctly set up network routing to your (private) RDS database but you don't have a network route to the Secrets Manager service endpoint, hence that request times out. If so, you'd need a VPC Endpoint for Secrets Manager or a NAT (or NAT Gateway) with routing to the public internet.