AWS CloudFormation: How to get stream ARN of DynamoDB GlobalTable replica?

Question

Objective

I want to deploy via CloudFormation a DynamoDB global table in the eu-west-1 region, with a replica in ap-northeast-1. In each region, I want to have a Lambda to handle events from the corresponding region replica.

I'm interested in avoiding manually hardcoding the StreamArn value in ap-northeast-1.

Failed approach

I came up with the following template (only relevant details shown for clarity):

Conditions:
  IsMainRegionCondition: !Equals [ !Ref AWS::Region, "eu-west-1" ]

Resources:
  MyDynamoDB:
    Condition: IsMainRegionCondition
    Type: AWS::DynamoDB::GlobalTable
    Properties:
      Replicas:
        - Region: eu-west-1
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: true
        - Region: ap-northeast-1
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: true

  DynamoToESFunction:
    Type: AWS::Serverless::Function
    Properties:
      Events:
        RawDynamoDBEvent:
          Type: DynamoDB
          Properties:
            Stream: !GetAtt MyDynamoDB.StreamArn

This works fine in eu-west-1, but in ap-northeast-1, I get: Template format error: Unresolved resource dependencies [MyDynamoDB] in the Resources block of the template .

Question

I do understand that the approach above does not work because the resource MyDynamoDB is not created in ap-northeast-1. However, I'm wondering what the best way to get this setup working without the need to hardcode the ARN is.

My current thoughts are to split my CloudFormation stack into 2: First deploy a stack with the DynamoDB table, and then reference (without hardcoding) the StreamArn of the replica in the second.

From the Global DynamoDB CloudFormation docs (under Return values - StreamArn): The StreamArn returned is that of the replica in the region the stack is deployed to . So in my case that would mean its only possible to access the value in eu-west-1.

Is it possible to reference the StreamArn value of the replica region (ap-northeast-1) without having to hardcode its value?

Thanks in advance.

Answer 1

With the help of this blog entry , I came up with the following solution using a custom resource.

In short, I split my stack into 2: In one I deploy the DynamoDB::GlobalTable , along with a Serverless::Function which will be in charge of getting the ARN for us. In the other, I have the rest of components.

The template for the first stack looks as follows:

AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31

Parameters:
  DynamoTableName:
    Type: "String"
    Default: TableNameExample
  MainRegion:
    Type: "String"
    Default: "eu-west-1"


Conditions:
  IsMainRegionCondition: !Equals [ !Ref AWS::Region, !Ref MainRegion ]


Resources:
  DynamoTable:
    Condition: IsMainRegionCondition  # Note: Deployed only in one region, replicated in others
    Type: AWS::DynamoDB::GlobalTable
    Properties:
      TableName: !Ref DynamoTableName
      BillingMode: PAY_PER_REQUEST
      KeySchema:
        - AttributeName: entity_id
          KeyType: HASH
        - AttributeName: language
          KeyType: RANGE
      AttributeDefinitions:
        - AttributeName: entity_id
          AttributeType: S
        - AttributeName: language
          AttributeType: S
      StreamSpecification:
        StreamViewType: NEW_AND_OLD_IMAGES
      Replicas:
        - Region: eu-west-1
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: true
        - Region: us-east-1
          PointInTimeRecoverySpecification:
            PointInTimeRecoveryEnabled: true

  GetGlobalTableStreamFunction:
    Type: AWS::Serverless::Function
    Properties:
      Environment:
        Variables:
          TABLE_NAME: !Ref DynamoTableName
      MemorySize: 256
      CodeUri: functions/dynamo_db_stream/
      Handler: app.lambda_handler
      Runtime: python3.7
      Policies:
        - Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action:
                - dynamodb:DescribeTable
              Resource: '*'

Outputs:
  GetGlobalTableStreamFunctionArn:
    Value: !GetAtt GetGlobalTableStreamFunction.Arn
    Export:
      Name: "get-global-table-stream-function-arn"

The code for the Lambda is the following (note you need to pack the crhelper dependency along the Lambda).

import logging
import os

import boto3
from crhelper import CfnResource

TABLE_NAME = os.environ["TABLE_NAME"]

logger = logging.getLogger(__name__)

helper = CfnResource(
    json_logging=False,
    log_level='DEBUG',
    boto_level='CRITICAL'
)

dynamo_db_client = boto3.client("dynamodb")


def lambda_handler(event, context):
    helper(event, context)


def get_table_stream_arn():
    logger.info(f"Getting stream ARN for table '{TABLE_NAME}'...")
    response = dynamo_db_client.describe_table(
        TableName=TABLE_NAME
    )

    logger.debug(f"Describe table response: {response}")
    stream_arn = response["Table"]["LatestStreamArn"]
    logger.info(f"ARN for table {TABLE_NAME}: {stream_arn}")
    return stream_arn


@helper.create
def create(event, context):
    logger.info("Received a 'Create' event")

    stream_arn = get_table_stream_arn()
    # This will make the stream ARN accessible via Cfn
    # `!GetAtt DynamoTableStreamArnGetter.StreamArn`
    helper.Data.update({"StreamArn": stream_arn})
    return stream_arn


@helper.update
def update(event, context):
    logger.info("Received an 'Update' event, doing nothing")


@helper.delete
def delete(event, context):
    logger.info("Received a 'Delete' event, doing nothing")

Then, in the second stack, we need to create the custom resource.

Resources:
  
  [...]

  DynamoTableStreamArnGetter:
    Type: 'Custom::DynamoTableStreamArnFunction'
    Version: '1.0'
    Properties:
      ServiceToken: !ImportValue "get-global-table-stream-function-arn"

Finally, you can get/reference the stream ARN in the replica regions (in the second stack template) through:

!GetAtt DynamoTableStreamArnGetter.StreamArn

Some notes about this solution:

1. I'm not sure if for this particular case, we need to return stream_arn in the create(...) function of the Lambda.
2. In the first template, I don't like the giving of permissions to describe all tables ( Resource: '*' ). However, there you cannot reference the DynamoTable resource since it will not exist in the replica regions. If anyone knows a better way to just restrict it to that table, please let me know.