GCP List vm instances and Get External IPs

Question

I am at wits end searching for several days now, I am no terraform expert but I cant seem to find anyway to get all the GCP Vm instance List of external ips inside of a Kubernetes cluster as i then need to use this list to add them to allowed list for Database access.

Can someone point me to an example or what

tried something like this but cant use * =\\

data "google_compute_instance" "instances" {
}

output "instance_ids" {
  value = "${data.google_compute_instance.instances.*.network_interface.0.access_config.0.nat_ip }"
}

Answer 1

I came up with one way to do this using Terraform, but it's slightly hacky as it involves a bit of string manipulation on some of the URLs due to the GKE cluster metadata returning a list of Instance Group Managers and not Instance Groups (despite what the attribute is labelled). However it does work for the cluster I tested it on.

If it helps, here's a Terraform example that outputs the external IPs for all the nodes compute instances:

provider "google" {
  version = "~> 2"
  project  = "my-project"
}

// GKE cluster details
data "google_container_cluster" "my_cluster" {
  name     = "my-cluster-name"
  location = "my-location"
} 

// GKE node instance group details
data "google_compute_instance_group" "node_instance_groups" {
    for_each = toset(data.google_container_cluster.my_cluster.node_pool[0].instance_group_urls)
    self_link = replace(each.key, "instanceGroupManagers", "instanceGroups")
}

// GKE node compute instance details
data "google_compute_instance" "nodes" {
    for_each = toset(flatten([for x in data.google_compute_instance_group.node_instance_groups : x.instances[*]]))
    self_link = each.key
}

// Return the external IPs for all GKE node instances
output "external_ips" {
    value = [for x in data.google_compute_instance.nodes : x.network_interface[0].access_config[0].nat_ip]
}

Answer 2

Rather than think about the reality that GKE nodes are implemented as Compute Engines, maybe think about GKE nodes in their Kubernetes concept. As a test, I created a cluster with 3 nodes and then ran:

kubectl get nodes -o wide

The result was

NAME                                STATUS   ROLES    AGE   VERSION           INTERNAL-IP   EXTERNAL-IP      OS-IMAGE                             KERNEL-VERSION   CONTAINER-RUNTIME
gke-c1-default-pool-5b6b460a-59nb   Ready    <none>   68s   v1.13.11-gke.14   10.128.0.42   35.222.104.41    Container-Optimized OS from Google   4.14.138+        docker://18.9.7
gke-c1-default-pool-5b6b460a-ggh9   Ready    <none>   68s   v1.13.11-gke.14   10.128.0.41   35.192.152.130   Container-Optimized OS from Google   4.14.138+        docker://18.9.7
gke-c1-default-pool-5b6b460a-j8nn   Ready    <none>   67s   v1.13.11-gke.14   10.128.0.40   104.197.68.223   Container-Optimized OS from Google   4.14.138+        docker://18.9.7

Please pay attention to the column called EXTERNAL-IP. I then compared these against the Compute Engine VM instance exposed public IP and found them to be identical.

Now that we see that examining the nodes using kubectl returns the information we desire, we can now potentially use a Terraform equivalent to this command. For example, the Kubernetes Provider .

Answer 3

My terraform knowledge is a little haisy but can you not do the following?

As mentioned on this page: https://www.terraform.io/docs/provisioners/local-exec.html

resource "gcp_instance" "web" {
  # ...

  provisioner "local-exec" {
    command = "echo gcloud compute instances list --format=\"value(External IP)\" --filter=\"name~'gke-$cluster_name-$pool_name*' --format=\"value(External_IP)\" "} >> public_ips.txt"
  }
}

then read it from the file?

Answer 4

The sortest solution to this problem is by using the gcloud command-line tool in conjunction with awk to manipulate the output.

With 2 simple commands, you get a nice formatted output.

First, calculate the number of nodes:

num_nodes=$(kubectl get nodes | awk 'END{print NR - 1}')

After

gcloud compute instances list | awk '{print $1,$5}' | tail -n $num_nodes

If you want to create a dynamic Ansible inventory, then the above command becomes:

gcloud compute instances list | awk '{print $1, "ansible_ssh_host="$5}' | tail -n $num_nodes