使用DNS的多云swarm集群服务发现

Question

I've managed to create a multi-cloud swarm cluster, with a leader node on GCP and a worker node on AWS. Deployment is running fine, but the only issue I've stumbled upon is the service discovery using DNS names.

For example, I have deployed a service called db on the leader node (GCP) and an app service on the worker node (AWS). They are using an overlay network defined as such:

networks:
  internal:
    attachable: true
    driver: overlay
    ipam:
      config:
        - subnet: 10.0.69.0/24

On the GCP network, I've opened the following ports:

• 2377 TCP port for cluster management communications;
• 7946 TCP&UDP ports for communication among nodes;
• 4789 for overlay network traffic.

On the AWS network, I've opened the following ports:

• 7946 TCP&UDP ports for communication among nodes;
• 4789 for overlay network traffic.

(source: https://docs.docker.com/engine/swarm/swarm-tutorial/ )

The db container gets the IP 10.0.69.8 and the app container gets the IP 10.0.69.21 . Containers are not able to ping each other, nor resolve the db and app DNS names. All I get is Destination Host Unreachable when I try to ping the other one.

Is my approach sketchy and I'm missing something? Or am I breaking something fundamental? Does a multi-cloud swarm cluster require already connected networks? And if so, is a VPN between GCP and AWS networks the only way?

Answer 1

If you want to create one swarm (cluster) in two different clouds you have to use VPN - to the best of my knowledge there's no way around it.

As you see the nodes can communicate using public IP's but the containers cannot (they can only use internal IP's) and your GCP & AWS networks aren't connected.

"Services or containers can only communicate across networks they are each connected to."

In such case DNS resolving isn't the issue here. Just the effect.

When you connect networks make sure the IP's of the GCP and AWS subnets aren't overlapping - otherwise you may face another issue and have to resort using double-NAT or some other very overcomplicated solutions.

It's best to keep it simple (read - better to use just one cloud) and keep your cluster there.