How can I join a Consul agent to a Consul cluster via a Consul server running in a different Fargate task with CloudFormation?

Question

I'm currently working as an intern and have to host a microservice application. I chose to use AWS ECS in combination with Fargate tasks to host a Consul Connect Service Mesh that provides service discovery, intentions, Mtls and so on for the application. My set-up is as follows:

• 1 Fargate task (in a service) with a Consul server container.
• x Fargate tasks (in a service) with a Consul agent, Consul Connect sidecar and a microservice container.

I am using CloudFormation to deploy that infrastructure automatically.

The problem:

I need to join the Consul agent running in one Fargate-task to the Consul cluster started in another Fargate task (task of the server) using CloudFormation. The problem I have with doing that is that I did not find any way to get the IP address to join the agent to the cluster.

Does anyone know how I can get the IP address of a Fargate task in CloudFormation or the best practice way of doing this?

If I am not mistaken you can only join a Consul agent to a Consul cluster using IP, DNS name or Cloud metadata. The first and second one I could not retrieve using CloudFormation and for the third one I found that it might not be possible (I could be wrong, but thats what I read so far).

Also I tried both the Consul agent -ui [...] -join and -retry-join flag, but neither one worked. I also tried creating an internal loadbalancer for the task with the Consul server from which I used the DNS name to try to join the Cluster, but that did not work either (I have never set-up a loadbalancer properly on AWS yet so I might have done that wrong). I tried that with the loadbalancer forwarding traffic to port 8500 (which was the wrong port I think) and afterwards with port 8301 (which I think was the right port). But I kept getting the message that there was no Consul Cluster on that address.

Can anyone tell me how I can proceed?

Thank you in advance!

Answer 1

Thanks to a very smart colleague of mine, I found that putting a load balancer (which I set up wrong earlier) in front of the ECS service with the Consul Server Fargate task solved my problem. The load balancer listener should listen on the SERF port 8301 tcp_udp and forward traffic to the service with that protocol and port.

  ConsulServerTargetGroup8301:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      HealthCheckEnabled: true
      HealthCheckIntervalSeconds: 30
      UnhealthyThresholdCount: 3
      HealthyThresholdCount: 3
      Name: ConsulServerTargetGroup8301
      Port: 8301
      Protocol: TCP_UDP
      TargetGroupAttributes:
        - Key: deregistration_delay.timeout_seconds
          Value: 60 # default is 300
      TargetType: ip
      VpcId: !Ref VPC

  ConsulServerTargetGroupListener8301:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - TargetGroupArn: !Ref ConsulServerTargetGroup8301
          Type: forward
      Port: 8301
      Protocol: TCP_UDP
      LoadBalancerArn: !Ref ConsulServerLoadbalancer
  
  ConsulServerLoadbalancer:
    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    Properties:
      Name: ConsulServerLoadbalancer
      IpAddressType: ipv4
      Type: network
      Scheme: internal
      SubnetMappings:
        - SubnetId: !Ref PrivateSubnet1a
        - SubnetId: !Ref PrivateSubnet1b 

You can then use the DNS name of the load balancer to join a Consul agent to the consul cluster using:

        Command:
        - !Sub >-
            consul agent 
            -ui 
            -data-dir consul/data 
            -advertise '{{ GetPrivateInterfaces | include "network" "${pVPCIpRange}" | attr "address" }}'
            -client 0.0.0.0
            -retry-join ${ConsulServerLoadbalancer.DNSName}