AWS cdk deploy --all fails to create ECS service

Question

new to the CDK and relatively new to AWS

The Issue

I'm following this tutorial which includes creating a fargate based private API, and accessing it on the public internet through an ec2 instance which is publicly exposed.

I'm picking through, minimally correcting various issues which gets everything running. It comes time to build with:

npm run build
cdk bootstrap
cdk synth FargateVpclinkStack
cdk deploy --all

Resulting in this being deployed:

I go out to eat and come back, and I'm still looking at the following:

[███████████████████████████████████████████████████████▊··] (52/54)

4:19:40 PM | CREATE_IN_PROGRESS   | AWS::CloudFormation::Stack                 | FargateVpclinkStack
4:23:17 PM | CREATE_IN_PROGRESS   | AWS::ECS::Service                          | bookService/Service

After waiting sufficiently long, the cloud formation was rolled back

10:43:36 PM | CREATE_FAILED        | AWS::ECS::Service                          | bookService/Service
Resource timed out waiting for completion (RequestToken: f8b1d082-1ff3-5a84-938a-95a0ea2f0960)
10:43:45 PM | ROLLBACK_IN_PROGRESS | AWS::CloudFormation::Stack                 | FargateVpclinkStack
The following resource(s) failed to create: [bookService05FB6DBB]. Rollback requested by user.

FrgateVpclinkStack failed: Error: The stack named FargateVpclinkStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE
The stack named FargateVpclinkStack failed creation, it may need to be manually deleted from the AWS console: ROLLBACK_COMPLETE

I assume this means the ECS service bookService/Service failed to deploy, and thus the entire FrgateVpclinkStack was rolled back. I'm curious why that is, and how it can be fixed.

Code

This is the TypeScript used by the cdk to generate the cloud formation template for FrgateVpclinkStack, called fargate-vpclink-stack.ts in the tutorial

import * as cdk from "@aws-cdk/core";
import * as elbv2 from "@aws-cdk/aws-elasticloadbalancingv2";
import * as ec2 from "@aws-cdk/aws-ec2";
import * as ecs from "@aws-cdk/aws-ecs";
import * as ecr from "@aws-cdk/aws-ecr";
import * as iam from "@aws-cdk/aws-iam";
import * as logs from "@aws-cdk/aws-logs";
import * as apig from "@aws-cdk/aws-apigatewayv2";
import * as servicediscovery from "@aws-cdk/aws-servicediscovery";

export class FargateVpclinkStack extends cdk.Stack {
  
  //Export Vpclink and ALB Listener
  public readonly httpVpcLink: cdk.CfnResource;
  public readonly httpApiListener: elbv2.ApplicationListener;

  constructor(scope: cdk.Construct, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // VPC
    const vpc = new ec2.Vpc(this, "ProducerVPC");

    // ECS Cluster
    const cluster = new ecs.Cluster(this, "Fargate Cluster", {
      vpc: vpc,
    });

    // Cloud Map Namespace
    const dnsNamespace = new servicediscovery.PrivateDnsNamespace(
      this,
      "DnsNamespace",
      {
        name: "http-api.local",
        vpc: vpc,
        description: "Private DnsNamespace for Microservices",
      }
    );

    // Task Role
    const taskrole = new iam.Role(this, "ecsTaskExecutionRole", {
      assumedBy: new iam.ServicePrincipal("ecs-tasks.amazonaws.com"),
    });

    taskrole.addManagedPolicy(
      iam.ManagedPolicy.fromAwsManagedPolicyName(
        "service-role/AmazonECSTaskExecutionRolePolicy"
      )
    );

    // Task Definitions
    const bookServiceTaskDefinition = new ecs.FargateTaskDefinition(
      this,
      "bookServiceTaskDef",
      {
        memoryLimitMiB: 512,
        cpu: 256,
        taskRole: taskrole,
      }
    );

    const authorServiceTaskDefinition = new ecs.FargateTaskDefinition(
      this,
      "authorServiceTaskDef",
      {
        memoryLimitMiB: 512,
        cpu: 256,
        taskRole: taskrole,
      }
    );

    // Log Groups
    const bookServiceLogGroup = new logs.LogGroup(this, "bookServiceLogGroup", {
      logGroupName: "/ecs/BookService",
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    const authorServiceLogGroup = new logs.LogGroup(
      this,
      "authorServiceLogGroup",
      {
        logGroupName: "/ecs/AuthorService",
        removalPolicy: cdk.RemovalPolicy.DESTROY,
      }
    );

    const bookServiceLogDriver = new ecs.AwsLogDriver({
      logGroup: bookServiceLogGroup,
      streamPrefix: "BookService",
    });

    const authorServiceLogDriver = new ecs.AwsLogDriver({
      logGroup: authorServiceLogGroup,
      streamPrefix: "AuthorService",
    });

    // Amazon ECR Repositories
    const bookservicerepo = ecr.Repository.fromRepositoryName(
      this,
      "bookservice",
      "book-service"
    );

    const authorservicerepo = ecr.Repository.fromRepositoryName(
      this,
      "authorservice",
      "author-service"
    );

    // Task Containers
    const bookServiceContainer = bookServiceTaskDefinition.addContainer(
      "bookServiceContainer",
      {
        image: ecs.ContainerImage.fromEcrRepository(bookservicerepo),
        logging: bookServiceLogDriver,
      }
    );

    const authorServiceContainer = authorServiceTaskDefinition.addContainer(
      "authorServiceContainer",
      {
        image: ecs.ContainerImage.fromEcrRepository(authorservicerepo),
        logging: authorServiceLogDriver,
      }
    );

    bookServiceContainer.addPortMappings({
      containerPort: 80,
    });

    authorServiceContainer.addPortMappings({
      containerPort: 80,
    });

    //Security Groups
    const bookServiceSecGrp = new ec2.SecurityGroup(
      this,
      "bookServiceSecurityGroup",
      {
        allowAllOutbound: true,
        securityGroupName: "bookServiceSecurityGroup",
        vpc: vpc,
      }
    );

    bookServiceSecGrp.connections.allowFromAnyIpv4(ec2.Port.tcp(80));

    const authorServiceSecGrp = new ec2.SecurityGroup(
      this,
      "authorServiceSecurityGroup",
      {
        allowAllOutbound: true,
        securityGroupName: "authorServiceSecurityGroup",
        vpc: vpc,
      }
    );

    authorServiceSecGrp.connections.allowFromAnyIpv4(ec2.Port.tcp(80));

    // Fargate Services
    const bookService = new ecs.FargateService(this, "bookService", {
      cluster: cluster,
      taskDefinition: bookServiceTaskDefinition,
      assignPublicIp: false,
      desiredCount: 2,
      securityGroup: bookServiceSecGrp,
      cloudMapOptions: {
        name: "bookService",
        cloudMapNamespace: dnsNamespace,
      },
    });

    const authorService = new ecs.FargateService(this, "authorService", {
      cluster: cluster,
      taskDefinition: authorServiceTaskDefinition,
      assignPublicIp: false,
      desiredCount: 2,
      securityGroup: authorServiceSecGrp,
      cloudMapOptions: {
        name: "authorService",
        cloudMapNamespace: dnsNamespace,
      },
    });

    // ALB
    const httpApiInternalALB = new elbv2.ApplicationLoadBalancer(
      this,
      "httpapiInternalALB",
      {
        vpc: vpc,
        internetFacing: false,
      }
    );

    // ALB Listener
    this.httpApiListener = httpApiInternalALB.addListener("httpapiListener", {
      port: 80,
      // Default Target Group
      defaultAction: elbv2.ListenerAction.fixedResponse(200),
    });

    // Target Groups
    const bookServiceTargetGroup = this.httpApiListener.addTargets(
      "bookServiceTargetGroup",
      {
        port: 80,
        priority: 1,
        healthCheck: {
          path: "/api/books/health",
          interval: cdk.Duration.seconds(30),
          timeout: cdk.Duration.seconds(3),
        },
        targets: [bookService],
        pathPattern: "/api/books*",
      }
    );

    const authorServiceTargetGroup = this.httpApiListener.addTargets(
      "authorServiceTargetGroup",
      {
        port: 80,
        priority: 2,
        healthCheck: {
          path: "/api/authors/health",
          interval: cdk.Duration.seconds(30),
          timeout: cdk.Duration.seconds(3),
        },
        targets: [authorService],
        pathPattern: "/api/authors*",
      }
    );

    //VPC Link
    this.httpVpcLink = new cdk.CfnResource(this, "HttpVpcLink", {
      type: "AWS::ApiGatewayV2::VpcLink",
      properties: {
        Name: "http-api-vpclink",
        SubnetIds: vpc.privateSubnets.map((m) => m.subnetId),
      },
    });
  }
}

This is all being done in cloud 9, with cdk version 1.105.0 (build 4813992) . My package.json has the following:

{
  "name": "cdk",
  "version": "0.1.0",
  "bin": {
    "cdk": "bin/cdk.js"
  },
  "scripts": {
    "build": "tsc",
    "watch": "tsc -w",
    "test": "jest",
    "cdk": "cdk"
  },
  "devDependencies": {
    "@aws-cdk/assert": "1.101.0",
    "@aws-cdk/aws-apigatewayv2": "1.101.0",
    "@aws-cdk/core": "1.101.0",
    "@aws-cdk/aws-ec2": "1.101.0",
    "@aws-cdk/aws-ecr": "1.101.0",
    "@aws-cdk/aws-ecs": "1.101.0",
    "@aws-cdk/aws-elasticloadbalancingv2": "1.101.0",
    "@aws-cdk/aws-iam": "1.101.0",
    "@aws-cdk/aws-logs": "1.101.0",
    "@aws-cdk/aws-servicediscovery": "1.101.0",
    "@types/jest": "^26.0.10",
    "@types/node": "10.17.27",
    "jest": "^26.4.2",
    "ts-jest": "^26.2.0",
    "aws-cdk": "1.101.0",
    "ts-node": "^9.0.0",
    "typescript": "~3.9.7"
  },
  "dependencies": {
    "@aws-cdk/core": "1.101.0",
    "source-map-support": "^0.5.16"
  }
}

all the code from the entire tutorial can be found at this github link

Answer 1

The timeout was due to a misnamed ECR which the bookService was attempting to access. To generalize this answer a bit, if there's a timeout it may be good to record which resources timed out and sanity check all the constituent elements.