Disable external IP on GKE nodes

Question

I'm using Jupyterhub + Kubernetes to provide a hosted development environment for a large programming class (>100 students). It's running on top of GKE with autoscaling enabled. As additional students log in, more nodes are dynamically added to the pool to handle the increased demand.

I'm running into an issue where the node pool is exhausting the project quota of external IPs, effectively limiting the size of the pool to 8 concurrent nodes. The exact error is this one . The nodes sit behind a reverse proxy for communicating with end user; as far as I can tell, the only use of these public IPs is to enable direct SSH into each individual node. I don't need or even want this functionality, since it presents a unnecessary attack surface.

How can I disable the automatic assignment of ephemeral IPs to these worker nodes? There must be a way since the docs for GKE suggest that autoscaling can grow up to something like 1000 nodes. I don't see how this could be possible if they are all subject to the same tiny external IP quota.

Answer 1

The solution you're looking for is simply increasing your quotas in console for your GCP project (IAM & admin -> Quotas). It's just a few clicks and usually takes only a few minutes for them to get approved.

Right now it's not possible to create GKE nodes without public IPs. Even if it were, it wouldn't help you as you'd just hit other quotas (cpu/disk), so also raise those.

Answer 2

1. Nodes that belong to a private cluster do not get assigned an external IP address.
2. Inspect the quota limits in the zone where your cluster is hosted. Pro tip: It is likely that if you are hitting these limits at low numbers (under 500 IPs/CPUs), then the zone is overutilized and you are probably paying measurably more than what the price would be in less utilised zones.

Answer 3

Unfortunately external IP addresses are necessary for each node to connect to the master. In order to avoid future problems you should request quota for external IP addresses and CPUs, so the cluster can autoscale.

Regarding the surface attack, you can check on your firewall rules that when a cluster is created, new rules are created and by default SSH to the nodes is allowed only to a range of IP addresses.