aws ECS, ECS instance is not registered to ALB target group

Question

I create ECS service and it runs 1 ecs instance and I can see the instance is registered as a target of the load balancer.

Now I trigger a Auto Scaling Group (by just incrementing desired instance count) to launch a new instance.

The instance is launched and added to the ECS cluster. (I can see it on ECS instances tab) But the instance is not added to the ALB target. (I expect to see 2 instances in the following image, but I only see 1)

I can edit AutoScalingGroup 's target group like the following

Then I see the following.

But the health check fails. It seems the 80 port is not reachable. Although I have port 80 open for public in the security group for the instance. (Also, instance created from ecs service uses dynamic port mapping but instance created by ALS does not)

So AutoScalingGroup can launch new instance but my load balancer never gives traffic to the new instance.

I did try https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-unhealthy-checks-ecs/?nc1=h_ls and it shows I can connect to port 80 from host to the docker container by something like curl -v http://${IPADDR}/health .

So it must be the case that there's something wrong with host port 80 (load balancer can't connect to it). But it is also the case the security group setting is not wrong, because the working instance and this non working instance is using the same SG.

• Edit

Because I used dynamic mapping, my webserver is running on some random port. As you can see the instance started by ecs service has registered itself to target group with random port.
However instance started by ALB has registered itself to target group with port 80.

Answer 1

It seems you know the root cause, which is that port 80 is failing the health check and thats why it is never added to ALB. Here is what you can try

First, check that your service is listening on port 80 on the new host. You can use command like netcat

nv -v localhost 80

Once you know that the service is listening, the recommended way to allow your ALB to connect to your host is to add a Security group inbound rule for your instance to allow traffic from your ALB security group on port 80

Answer 2

The instance will not be added to the target group if it's not healthy. So you need to fix the health check first.

From your first instance, your mapped port is 32769 so I assume if this is the same target group and if it is the same application then the port in new instance should be 32769 .

When you curl the IP endpoint curl -I -v http://${IPADDR}/health. is the HTTP status code was 200 , if it is 200 then it should be healthy if it's not 200 then update the backend http-status code or you can update health check HTTP status code.

I assume that you are also running ECS in both instances, so ECS create target group against each ECS services, are you running some mix services that you need target group in AS group? if you are running dynamic port then remove the health check path to traffic port.

Now if we look the offical possible causes for 502 bad Gateway

Dynamic port mapping is a feature of container instance in Amazon Elastic Container Service (Amazon ECS)

Dynamic port mapping with an Application Load Balancer makes it easier to run multiple tasks on the same Amazon ECS service on an Amazon ECS cluster.

With the Classic Load Balancer, you must statically map port numbers on a container instance. The Classic Load Balancer does not allow you to run multiple copies of a task on the same instance because the ports conflict. An Application Load Balancer uses dynamic port mapping so that you can run multiple tasks from a single service on the same container instance.

Your created target group will not work with dynamic port, you have to bind the target group with ECS services.

dynamic-port-mapping-ecs

HTTP 502: Bad Gateway Possible causes:

• The load balancer received a TCP RST from the target when attempting to establish a connection.

• The load balancer received an unexpected response from the target, such as "ICMP Destination unreachable (Host unreachable)", when attempting to establish a connection. Check whether traffic is allowed from the load balancer subnets to the targets on the target port.

• The target closed the connection with a TCP RST or a TCP FIN while the load balancer had an outstanding request to the target. Check whether the keep-alive duration of the target is shorter than the idle timeout value of the load balancer.

• The target response is malformed or contains HTTP headers that are not valid.

• The load balancer encountered an SSL handshake error or SSL handshake timeout (10 seconds) when connecting to a target.

• The deregistration delay period elapsed for a request being handled by a target that was deregistered. Increase the delay period so that lengthy operations can complete. http-502-issues