AWS Blue/Green CodeDeploy to ECS install lifecycle event timesout

Question

As the title suggests, the blue/green deployment for ecs never finishes because the install lifecycle event never finishes and timesout.

This is the picture showing that: The appspec file:

version: 0.0 
Resources: 
  - TargetService: 
      Type: AWS::ECS::Service 
      Properties: 
        TaskDefinition: <TASK_DEFINITION> 
        LoadBalancerInfo: 
          ContainerName: "WordpressContainer" 
          ContainerPort: 80 

The taskdef file:

{ 
    "executionRoleArn": "arn:aws:iam::336636872471:role/WordpressPipelineExecutionRole", 
    "containerDefinitions": [ 
        { 
            "name": "WordpressContainer", 
            "image": "<IMAGE1_NAME>", 
            "essential": true, 
            "portMappings": [ 
                { 
                    "hostPort": 80, 
                    "protocol": "tcp", 
                    "containerPort": 80 
                } 
            ] 
        } 
    ], 
    "requiresCompatibilities": [ 
        "FARGATE" 
    ], 
    "networkMode": "awsvpc", 
    "cpu": "256", 
    "memory": "512", 
    "family": "wordpress" 
} 

I am pushing a bare-bones wordpress docker image to ECR which triggers a pipeline but it stucks on CodeDeploy .

Any ideas what is happening? How am I even supposed to debug that?

PS it timed-out in 60 minutes with the message:

The deployment timed out while waiting for the replacement task set to become healthy. This time out period is 60 minutes.

Answer 1

I would check the target groups's health check since it is waiting for a replacement task to become healthier. Is your current deployment of ECS targets HEALTHY? If they are not, the ALB will be trying to bounce these containers to try and refresh them to have a health check passed. Also, does your CodeDeplot have access to deploy to ECR?

Answer 2

I'm running into a similar issue. I looked at the cloud logs and didn't see anything obvious. I based my architecture off of https://pypi.org/project/cloudcomponents.cdk-blue-green-container-deployment/

Below are links to images of the results

Code Deploy Failure

Lifecycle events

Target Group 1

Target Group 2

appspec.yaml

version: 0.0
Resources:
  - TargetService:
      Type: AWS::ECS::Service
      Properties:
        TaskDefinition: <TASK_DEFINITION>
        LoadBalancerInfo:
          ContainerName: "sample-website"
          ContainerPort: 80

buildspec.yml

version: 0.2

phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws --version
      - aws ecr get-login-password | docker login --username AWS --password-stdin $REPOSITORY_URI
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:=latest}
      - echo Setting environment variables...
      - echo $EXECUTION_ROLE_ARN
      - echo $FAMILY
      - sed -i "s|SED_REPLACE_EXECUTION_ROLE_ARN|$EXECUTION_ROLE_ARN|g" taskdef.json
      - sed -i "s|SED_REPLACE_FAMILY|$FAMILY|g" taskdef.json
      - cat taskdef.json
  build:
    commands:
      - echo Docker build and tagging started on `date`
      - docker build -t $REPOSITORY_URI:latest -t $REPOSITORY_URI:$IMAGE_TAG -f Dockerfile .
      - echo Docker build and tagging completed on `date`
  post_build:
    commands:
- echo Pushing the Docker images to container registry...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - echo Writing image definitions file...
      - printf '{"ImageURI":"%s"}' $REPOSITORY_URI:$IMAGE_TAG > imageDetail.json
      - echo Build completed on `date`
artifacts:
  files:
    - "appspec.yaml"
    - "taskdef.json"
  secondary-artifacts:
    ManifestArtifact:
      files:
        - appspec.yaml
        - taskdef.json
    ImageArtifact:
      files:
        - imageDetail.json

taskdef.json

{
  "executionRoleArn": "SED_REPLACE_EXECUTION_ROLE_ARN",
  "containerDefinitions": [
    {
      "name": "sample-website",
      "image": "<IMAGE1_NAME>",
      "essential": true,
      "portMappings": [
        {
          "hostPort": 80,
          "protocol": "tcp",
          "containerPort": 80
        }
      ]
    }
  ],
  "requiresCompatibilities": ["FARGATE"],
  "networkMode": "awsvpc",
  "cpu": "256",
  "memory": "512",
  "family": "SED_REPLACE_FAMILY"
}

from aws_cdk import (
    aws_ec2 as ec2,
    Stack,
    aws_ecs as ecs,
    aws_ecs_patterns as ecs_patterns,
    aws_codecommit as codecommit,
    aws_ecr as ecr,
    aws_codepipeline as codepipeline,
    aws_codepipeline_actions as pipeline_actions,
    aws_codebuild as codebuild,
    aws_codedeploy as codedeploy,
    aws_elasticloadbalancingv2 as ecb,
    Duration
)

from cloudcomponents.cdk_blue_green_container_deployment import (
    EcsDeploymentGroup,
    EcsService,
    DummyTaskDefinition
)
from os import path
from constructs import Construct

class Ab3ArchitectureStack(Stack):

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        source_artifact = codepipeline.Artifact(artifact_name='SourceArtifact')
        image_artifact = codepipeline.Artifact(artifact_name='ImageArtifact')
        manifest_artifact = codepipeline.Artifact(artifact_name='ManifestArtifact')

        ab3_code_repo = codecommit.Repository(self, 'AB3_Code_Repository',
            repository_name="AB3_NginxApp", #this name is what's seen in CodeCommit
            #code=codecommit.Code.from_directory(path.abspath( "/home/ec2-user/environment/AB3_NginxApp"))
        )

        ab3_ecr_repo = ecr.Repository(self, 'AB3_ECR_Repository',
            image_scan_on_push=True
        )

        #Ran into a build error so created a policy(AB3ElasticContainerRegistry) and attached it to Ab3ArchitectureStack-AB3CodeBuildRole
        #Fixed token issue, but then neded to attach policy to Ab3ArchitectureStack-AB3PipelineBuildAB3BuildCode
        ab3_build = codebuild.Project(self, 'AB3_Code_Build',
            source=codebuild.Source.code_commit(
                repository=ab3_code_repo
            ),
            environment=codebuild.BuildEnvironment(
                build_image=codebuild.LinuxBuildImage.STANDARD_5_0,
                compute_type=codebuild.ComputeType.SMALL,
                privileged=True,
                environment_variables={
                    'REPOSITORY_URI' : codebuild.BuildEnvironmentVariable(value=ab3_ecr_repo.repository_uri),
                    'FAMILY' : codebuild.BuildEnvironmentVariable(value='AB3-blue-green-family'),
                    #The below is a temporary fix and won't work in other enivronments
                    'EXECUTION_ROLE_ARN' : codebuild.BuildEnvironmentVariable(value='arn:aws:iam::460994089204:role/Admin')
                }
            )
        )

        ab3_vpc = ec2.Vpc(self, 'AB3Vpc', max_azs=2)     # default is all AZs in region

        ab3_cluster = ecs.Cluster(self, 'AB3Cluster', vpc=ab3_vpc)

        ab3_load_balancer = ecb.ApplicationLoadBalancer(self, 'AB3LoadBalancer',
            vpc=ab3_vpc,
            internet_facing=True
        )

        ab3_prod_listener = ab3_load_balancer.add_listener('ProductionListener',
            port=80
        )

        ab3_test_listener = ab3_load_balancer.add_listener('TestListener',
            port=8080
        )

        ab3_prod_tgt_group = ecb.ApplicationTargetGroup(self, 'ProdTargetGoup',
            port=80,
            target_type= ecb.TargetType.IP,
            vpc=ab3_vpc
        )

        ab3_prod_listener.add_target_groups('AddProdTgtGroup',
            target_groups=[ab3_prod_tgt_group]
        )

        ab3_test_tgt_group = ecb.ApplicationTargetGroup(self, 'TestTargetGoup',
            port=8080,
            target_type= ecb.TargetType.IP,
            vpc=ab3_vpc
        )

        ab3_test_listener.add_target_groups('AddTestTgtGroup',
            target_groups=[ab3_test_tgt_group]
        )

        ab3_ecs_service = EcsService(self, "AB3ECSService",
            cluster=ab3_cluster,
            service_name='AB3-Service',
            desired_count=2,
            task_definition= DummyTaskDefinition(self, 'DummyTaskDef', #should be replaced by CodeDeploy in CodePipeline
                image='nginx',
                family='AB3-blue-green-family'
            ),
            test_target_group=ab3_test_tgt_group,
            prod_target_group=ab3_prod_tgt_group
        )

        ab3_ecs_service.connections.allow_from(ab3_load_balancer, ec2.Port.tcp(80))
        ab3_ecs_service.connections.allow_from(ab3_load_balancer, ec2.Port.tcp(8080))

        #ab3_deployment_group = codedeploy.IEcsDeploymentGroup() TypeError Protocols can't be instantiated
        ab3_deployment_group = EcsDeploymentGroup(self, "AB3DeployGroup",
            application_name='AB3-Application',
            deployment_group_name='AB3-Deployment-Group',
            ecs_services=[ab3_ecs_service],
            target_groups=[ab3_prod_tgt_group,ab3_test_tgt_group],
            prod_traffic_listener= ab3_prod_listener,
            test_traffic_listener=ab3_test_listener,
            termination_wait_time=Duration.minutes(15)
        )

        pipeline = codepipeline.Pipeline(self, "AB3Pipeline",
        stages=[
            codepipeline.StageProps(
                stage_name="Source",
                actions=[
                    pipeline_actions.CodeCommitSourceAction(
                        repository=ab3_code_repo,
                        branch='main',
                        output=source_artifact,
                        action_name='AB3Source'
                    )
                ] 
            ),
            codepipeline.StageProps(
                stage_name="Build",
                actions=[
                    pipeline_actions.CodeBuildAction(
                        input=source_artifact,
                        project=ab3_build,
                        action_name='AB3Build',
                        outputs=[image_artifact, manifest_artifact] #second output needs to be specified in buildspec
                        #outputs=[image_artifact]
                    )
                ]
            ),
            codepipeline.StageProps(
                stage_name="Deploy",
                actions=[
                    pipeline_actions.CodeDeployEcsDeployAction(
                        action_name="AB3Deploy",
                        deployment_group=ab3_deployment_group,
                        #app_spec_template_input=image_artifact,
                        #task_definition_template_input=image_artifact
                        app_spec_template_input=manifest_artifact,
                        task_definition_template_input=manifest_artifact,
                        container_image_inputs=[pipeline_actions.CodeDeployEcsContainerImageInput(
                            input=image_artifact,
                            task_definition_placeholder='IMAGE1_NAME'
                        )]
                    )
                ]
            )
        ]
        ) 

        '''
        ab3_ecs_application = codedeploy.EcsApplication(self, "AB3_Application",
            application_name="AB3_NGINX_Application"
        )

        ab3

        ab3_task_definition = ecs.FargateTaskDefinition(self, 'AB3_Task_Definition',
            cpu=256,
            memory_limit_mib=1024
            #task_role= ???,
            #execution_role= ????
        )

        ab3_task_definition.add_container('AB3_Container', 
            image= ecs.ContainerImage.from_ecr_repository(ab3_ecr_repo)

        )

        # Provide a Stage when creating a pipeline

        ecs_patterns.ApplicationLoadBalancedFargateService(self, "AB3Service",
            cluster=ab3_cluster,            # Required
            cpu=512,                    # Default is 256
            desired_count=6,            # Default is 1
            task_image_options=ecs_patterns.ApplicationLoadBalancedTaskImageOptions(
                image=ecs.ContainerImage.from_registry("amazon/amazon-ecs-sample")),
                #image=ecs.ContainerImage.from_ecr_repository(ab3_ecr_repo)),
            memory_limit_mib=2048,      # Default is 512
            public_load_balancer=True  # Default is True
        )
'''