AWS ELB: 503 Service Temporarily Unavailable

Question

I am trying to deploy microservices to AWS ECS following this example repo . Here, a load balancer is utilized for networking between different docker services. I have adjusted the cloudformation templates to my needs and have only deployed one service yet, the webserver. I want public access to the web interface and add other services this service is talking to subsequently. However, I currently struggle to get the webserver going using the URL of the load balancer. If I go directly to the EC2 instance using the public IP, I get access to the interface. However, if I go to the load balancer DNS, I get 503 Service Temporarily Unavailable . I have checked the AWS docs , but the webservice target group shows a registered target (the EC2 instance) with status healthy . What am I missing?

Parameters:
  EnvironmentName:
    Description: An environment name that will be prefixed to resource names
    Type: String

  VPC:
    Type: AWS::EC2::VPC::Id
    Description: Choose which VPC the Application Load Balancer should be deployed to

  Subnets:
    Description: Choose which subnets the Application Load Balancer should be deployed to
    Type: AWS::EC2::Subnet::Id

  PublicSubnet:
    Description: Choose which public subnet the EC2 instance should be deployed to
    Type: AWS::EC2::Subnet::Id

Resources:

  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${EnvironmentName}-loadbalancer
      GroupDescription: Access to the load balancer that sits in front of ECS
      VpcId: !Ref VPC
      SecurityGroupIngress:
        # Allow access from anywhere to our ECS services
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName}-LoadBalancers

  LoadBalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: !Ref EnvironmentName
      Subnets:
        - !Ref Subnets
      SecurityGroups:
        - !Ref LoadBalancerSecurityGroup
      Tags:
        - Key: Name
          Value: !Ref EnvironmentName

  LoadBalancerListener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      LoadBalancerArn: !Ref LoadBalancer
      Port: 80
      Protocol: HTTP
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref DefaultTargetGroup

  DefaultTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: !Sub ${EnvironmentName}-default
      VpcId: !Ref VPC
      Port: 80
      Protocol: HTTP

  ECSHostSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${EnvironmentName}-ecs-hosts
      GroupDescription: Access to the ECS hosts and the tasks/containers that run on them
      VpcId: !Ref VPC
      SecurityGroupIngress:
        # Only allow inbound access to ECS from the ELB
        - SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
          IpProtocol: -1
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName}-ECS-Hosts

  ECSCluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Ref EnvironmentName

  ECSRole:
    Type: AWS::IAM::Role
    Properties:
      Path: /
      RoleName: !Sub ${EnvironmentName}-ecs-role
      AssumeRolePolicyDocument:
        Statement:
          - Effect: Allow
            Principal:
              Service: ec2.amazonaws.com
            Action: 'sts:AssumeRole'
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceforEC2Role'
        - 'arn:aws:iam::aws:policy/service-role/AmazonEC2RoleforSSM'
        - 'arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy'

  ECSInstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
        - !Ref ECSRole

  EC2Webserver:
    Type: AWS::EC2::Instance
    Properties:
      AvailabilityZone: eu-central-1a
      ImageId: !Ref ECSAMI
      InstanceType: !Ref InstanceType
      IamInstanceProfile: !Ref ECSInstanceProfile
      UserData:
        Fn::Base64:
          !Sub |
          #!/bin/bash
          # Add to cluster:
          echo ECS_CLUSTER=${ECSCluster} >> /etc/ecs/ecs.config
          echo ECS_ENABLE_AWSLOGS_EXECUTIONROLE_OVERRIDE=true >> /etc/ecs/ecs.config
      SecurityGroupIds:
        - !Ref ECSHostSecurityGroup
      SubnetId: !Ref PublicSubnet
      Tags:
        - Key: Name
          Value: !Sub ${EnvironmentName}

  Service:
    Type: AWS::ECS::Service
    DependsOn: ListenerRule
    Properties:
      Cluster: !Ref Cluster
      Role: !Ref ServiceRole
      DesiredCount: !Ref DesiredCount
      TaskDefinition: !Ref TaskDefinitionWebserver
      LoadBalancers:
        - ContainerName: !Sub ${EnvironmentName}-webserver
          ContainerPort: 8080
          TargetGroupArn: !Ref TargetGroup

  TaskDefinitionWebserver:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub ${EnvironmentName}-webserver
      ContainerDefinitions:
        - Name: !Sub ${EnvironmentName}-webserver
          Essential: true
          Image: !Ref Image
          Memory: 512
          PortMappings:
            - ContainerPort: 8080
              HostPort: 80

  TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      Name: !Sub ${EnvironmentName}-webserver
      VpcId: !Ref VPC
      Port: 80
      Protocol: HTTP
      Matcher:
        HttpCode: 200-299
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: /health
      HealthCheckProtocol: HTTP
      HealthCheckTimeoutSeconds: 10
      HealthyThresholdCount: 5

  ListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 1
      Conditions:
        - Field: path-pattern
          Values:
            - /
      Actions:
        - TargetGroupArn: !Ref TargetGroup
          Type: forward

  ServiceRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub ecs-service-${AWS::StackName}
      Path: /
      AssumeRolePolicyDocument: |
        {
            "Statement": [{
                "Effect": "Allow",
                "Principal": { "Service": [ "ecs.amazonaws.com" ]},
                "Action": [ "sts:AssumeRole" ]
            }]
        }
      Policies:
        - PolicyName: !Sub ecs-service-${AWS::StackName}
          PolicyDocument:
            {
              "Version": "2012-10-17",
              "Statement":
                [
                {
                  "Effect": "Allow",
                  "Action":
                    [
                      "ec2:AuthorizeSecurityGroupIngress",
                      "ec2:Describe*",
                      "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                      "elasticloadbalancing:Describe*",
                      "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
                      "elasticloadbalancing:DeregisterTargets",
                      "elasticloadbalancing:DescribeTargetGroups",
                      "elasticloadbalancing:DescribeTargetHealth",
                      "elasticloadbalancing:RegisterTargets"
                    ],
                  "Resource": "*"
                }
                ]
            }

Outputs:

  WebsiteServiceUrl:
    Description: The URL endpoint for the website service
    Value: !Join ["", [!GetAtt LoadBalancer.DNSName, "/"]]

Answer 1

Thanks everyone! I finally figured it out, what I had to do was adjusting the path as my service redirects on / . So I only changed the listener rule using a wildcard:

ListenerRule:
    Type: AWS::ElasticLoadBalancingV2::ListenerRule
    Properties:
      ListenerArn: !Ref LoadBalancerListener
      Priority: 1
      Conditions:
        - Field: path-pattern
          Values:
            - [/*]
      Actions:
        - TargetGroupArn: !Ref TargetGroup
          Type: forward

Answer 2

SG egress rules

It looks the Security Groups (SG) has no egress rule defined. Both ALB and EC2.

  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${EnvironmentName}-loadbalancer
      GroupDescription: Access to the load balancer that sits in front of ECS
      VpcId: !Ref VPC
      SecurityGroupIngress:
        # Allow access from anywhere to our ECS services
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1

I believe when you look at the outbound rules of the SG in the EC2 console, there would be no rule. If this is true, then the cause would be that traffic can come in to port 80 of ALB, but it cannot go out from ALB,

This is my theory. So please add a egress rule to verify?

  LoadBalancerSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${EnvironmentName}-loadbalancer
      GroupDescription: Access to the load balancer that sits in front of ECS
      VpcId: !Ref VPC
      SecurityGroupIngress:
        # Allow access from anywhere to our ECS services
        - CidrIp: 0.0.0.0/0
          IpProtocol: -1
      SecurityGroupEgress:
      - IpProtocol: tcp
        FromPort: 80
        ToPort: 80
        CidrIp: 0.0.0.0/0   # <--- Maybe better change to VPC CIDR or ECS/EC2 subnet CIDR rather than any IP.

Regarding EC2, because SG is stateful, traffic can go through port 80 to reach the port 8080 of the docker container, and the response can go back through the SG because the SG knows it is the response of the incoming connection.

Whereas for ALB, the incoming connection from the Internet is terminated at ALB port 80, then a new outbound connection needs to be established to EC2 instance(s) port 80, hence needs an egress rule defined, if I am correct.

  ECSHostSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub ${EnvironmentName}-ecs-hosts
      GroupDescription: Access to the ECS hosts and the tasks/containers that run on them
      VpcId: !Ref VPC
      SecurityGroupIngress:
        # Only allow inbound access to ECS from the ELB
        - SourceSecurityGroupId: !Ref LoadBalancerSecurityGroup
          IpProtocol: -1
        - IpProtocol: tcp
          FromPort: 22
          ToPort: 22
          CidrIp: 0.0.0.0/0
        - IpProtocol: tcp
          FromPort: 80
          ToPort: 80
          CidrIp: 0.0.0.0/0

However, if the EC2 instances need to install packages or create an outbound connection, then the EC2 SG needs egress rules too.

IAM

Regarding the IAM role for the ECS service, there are pre-defined AWS managed role, so I suppose better to use them?

• AWS::ECS::Service

    "Role": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "AssumeRolePolicyDocument": {
          "Version": "2008-10-17",
          "Statement": [
            {
              "Sid": "",
              "Effect": "Allow",
              "Principal": {
                "Service": "ecs.amazonaws.com"
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "ManagedPolicyArns": ["arn:aws:iam::aws:policy/service-role/AmazonEC2ContainerServiceRole"]
      }
    }

As AWS has introduced Service Linked Role now, it should be even better to use it.

• Service-Linked Role for Amazon ECS

  Prior to the introduction of a service-linked role for Amazon ECS, you were required to create an IAM role for your Amazon ECS services which granted Amazon ECS the permission it needed. This role is no longer required, however it is available if needed. For more information, see Legacy IAM Roles for Amazon ECS.

"arn:aws:iam::${data.aws_caller_identity.current.account_id}:role/aws-service-role/ecs.amazonaws.com/AWSServiceRoleForECS"