无法从运行 Java 的 Lambda 检索 AWS 机密

Question

I am writing a AWS Lambda function that will handle requests from discord. I am storing my discord values in AWS SecretsManager and need to retrieve them so I can verify the requests.

I have my lambda set to time out after 30 seconds and when my lambda runs it times out and never get the secrets value. I ran the same code in my IDE and it was able to fetch the values. I even assumed the role of my lambda and was able to execute the code in my IDE as well. The lambda role has the following permissions:

secretsmanager:GetSecretValue, secretsmanager:DescribeSecret, secretsmanager:ListSecrets

I even gave the lambda role read permissions to the secret value. Would anyone know why its timing out in the lambda and not getting the value? Here is my lambda handler class.

public class HandlerStream implements RequestHandler<DiscordEventRequest, DiscordEventResponse> {
  DiscordSecrets discordSecrets;
  Gson gson = new GsonBuilder().setPrettyPrinting().create();

  @Override
  public DiscordEventResponse handleRequest(DiscordEventRequest event, Context context) {
    LambdaLogger logger = context.getLogger();

    logger.log(String.format("Received Event: %s", gson.toJson(event)));
    try {
      getDiscordSecrets(logger);

      DiscordEventResponse response = new DiscordEventResponse();
      switch (event.getJsonBody().getType()) {
        case 1:
          response.setType(1);
      }

      return response;
    } catch (Throwable ex) {
      logger.log("Exception occurred when executing lambda: " + ex.getMessage());
    }
   throw new RuntimeException("[UNAUTHORIZED] invalid state detected");
  }

  private DiscordSecrets getDiscordSecrets(LambdaLogger logger) {
    if (discordSecrets == null) {
      String tokenName = System.getenv("discordToken");

      logger.log("Token Name " + tokenName);

      AWSSecretsManager client = AWSSecretsManagerClientBuilder.standard().build();

      GetSecretValueRequest request = new GetSecretValueRequest().withSecretId(tokenName);

      GetSecretValueResult response = client.getSecretValue(request);

      String secret;

      logger.log("Got secret value");

      if (response.getSecretString() != null) {
        secret = response.getSecretString();
      } else {
        secret = new String(Base64.getDecoder().decode(response.getSecretString().getBytes(StandardCharsets.UTF_8)));
      }
      discordSecrets = gson.fromJson(secret, DiscordSecrets.class);
      logger.log("Discord Secrets " + gson.toJson(discordSecrets));
    }
    return discordSecrets;
  }
}

This is what is returned from the lambda (I changed it to 5 seconds for quicker time outs but it was timing out after 30 seconds as well)

2021-09-28T00:11:12.827-05:00   START RequestId: e5e211d0-9307-4609-9939-ff54f876e516 Version: $LATEST
2021-09-28T00:11:13.321-05:00   Received Event: { "jsonBody": { "type": 1, "version": 0 } }
2021-09-28T00:11:13.321-05:00   Token Name DiscordToken
2021-09-28T00:11:17.842-05:00   END RequestId: e5e211d0-9307-4609-9939-ff54f876e516
2021-09-28T00:11:17.842-05:00   REPORT RequestId: e5e211d0-9307-4609-9939-ff54f876e516 Duration: 5005.29 ms Billed Duration: 5000 ms Memory Size: 128 MB Max Memory Used: 102 MB Init Duration: 492.26 ms
2021-09-28T00:11:17.842-05:00   2021-09-28T05:11:17.833Z e5e211d0-9307-4609-9939-ff54f876e516 Task timed out after 5.01 seconds

It never hits the "Got secret value" nor does it throw an exception. All it does it wait until eventually the lambda times out.

Any help would be appreciated

Update I updated my lambda to use the v2 of the client as follows

private DiscordSecrets getDiscordSecrets(LambdaLogger logger) {
    if (discordSecrets == null) {
      String tokenName = System.getenv("discordToken");

      logger.log("Token Name " + tokenName);
      logger.log("Creating Client");
      SecretsManagerClient client = SecretsManagerClient.builder().region(Region.US_EAST_2).build();
      logger.log("Done creating client");
      String secret;

      GetSecretValueRequest request = GetSecretValueRequest.builder().secretId(tokenName).build();
      GetSecretValueResponse response = client.getSecretValue(request);

      logger.log("Got secret value");

      if (response.secretString() != null) {
        secret = response.secretString();
      } else {
        secret = new String(Base64.getDecoder().decode(response.secretBinary().asByteBuffer()).array());
      }
      discordSecrets = gson.fromJson(secret, DiscordSecrets.class);
      logger.log("Discord Secrets " + gson.toJson(discordSecrets));
    }
    return discordSecrets;
  }

It looks like it is hanging on trying to create the client. As it times out before Done creating client happens. IT also was hanging getting the client with the old method as well.

Answer 1

Troubleshooting steps:

1. Please specify the region while creating the client.

AWSSecretsManager client = AWSSecretsManagerClientBuilder.standard().withRegion(Regions.<YOUR_REGION>).build();

Please refer: https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/services/secretsmanager/AWSSecretsManagerClientBuilder.html

2. If this does not work and the lambda is still timing out define a VPC endpoint for the SecreteManager service.

Please refer: https://heathivie.com/2020/08/13/aws-lambda-and-secrets-manager-timeout/

Regards Amit

Answer 2

I figured out the issue. The problem was that Java Lambdas need time to spin up and then initialize the SecretsManager client which caused the time.

I ended up making the client static

public class HandlerStream implements RequestStreamHandler {
  static SecretsManagerClient client = SecretsManagerClient.builder().build();
  ...
}

Even after that I got an out of memory exception so I increased the memory of the lambda to 256 which seemed to solve the problem. Some reason the lambda is using 146 mb of memory upon execution. I dont know if this is due to how it handles java or not.

Answer 3

I ran into this problem when creating a lambda using Terraform. I'd not set a value for the timeout of the lambda and the default timeout value for Lambda execution is 3 seconds.

It was taking longer than 3 seconds to create the secretsmanager client using the Java SDK builder so my Lambda was hitting the execution timeout on every execution.

Facepalm.