amazon-ecs-agent is always restarting

Question

I have an autoscaling group on amazon, that spawns server instances.

My aws-cli config is:

[default]
s3 =
    signature_version = s3v4
    addressing_style = path
region = eu-central-1

My ecs-agent config gets read from s3 with aws-cli at instance bootup and is as follows:

ECS_CLUSTER=cluster-mongodb
ECS_ENGINE_AUTH_TYPE=dockercfg
ECS_ENGINE_AUTH_DATA={"domain.com":{"auth":"bsaewfgvewgf23g9hv3","email":""}}

Sadly the ecs-agent on the spawned instance is always restarting after 10 secs, instead of registering to the cluster "cluster-mongodb".

The error log is here:

2016-02-05T14:40:47Z [INFO] Starting Agent: Amazon ECS Agent - v1.7.1 (007985c)
2016-02-05T14:40:47Z [INFO] Loading configuration
2016-02-05T14:40:47Z [INFO] Checkpointing is enabled. Attempting to load state
2016-02-05T14:40:47Z [INFO] Loading state! module="statemanager"
2016-02-05T14:40:47Z [INFO] Detected Docker versions [1.17 1.18 1.19 1.20]
2016-02-05T14:40:47Z [INFO] Registering Instance with ECS
2016-02-05T14:41:07Z [ERROR] Could not register module="api client" err="RequestError: send request failed
caused by: Post https://ecs.eu-central-1.amazonaws.com/: net/http: request canceled while waiting for connection"
2016-02-05T14:41:07Z [ERROR] Error registering: RequestError: send request failed
caused by: Post https://ecs.eu-central-1.amazonaws.com/: net/http: request canceled while waiting for connection

Someone has seen this before? Is this maybe related to messed up security groups/LB config?

Thanks in advance

[EDIT] found the solution:

check, if the subnet is registered to the right route table (vpc -> route tables). in my case it was the wrong one.

Additionally my instances needed a public IP assigned by the launch configuration of the Autoscaling group, but that was special to my setup.

Answer 1

That message indicates your instance cannot do an HTTPS POST to the ecs.eu-central-1.amazonaws.com endpoint.

Make sure your security groups and/or VPC configuration allows outbound HTTPS to your ECS endpoint .

You can confirm endpoint connectivity from the instance to the endpoint the command line:

curl https://ecs.eu-central-1.amazonaws.com

That should return a message similar to the following:

<MissingAuthenticationTokenException>
  <Message>Missing Authentication Token</Message>
</MissingAuthenticationTokenException>

Answer 2

使用NAT实例，并让Internet通过VPC内部的私有VM。

Answer 3

Actually, ecs agent need internet access to connect with ecs endpoint ie https://ecs.eu-central-1.amazonaws.com .

Either ECS instance have pubic IP or they are using NAT instance/Gateway for public access.

see link https://forums.aws.amazon.com/message.jspa?messageID=781868 for more details