GCP - Regional GKE cluster with Network endpoint group NEG (with HTTP LoadBalancer and Cloud Armor)

Question

thanks to Gabriel Hodoroaga and his tutorial we have config with this flow in GCP:

Internet > HTTP Load Balancer > Network Endpoint Groups > GKE in one zone > ingress-nginx

But we need to switch GKE from zonal to regional. So I rebuild this config but lot of thinks I did manually via gcloud commands. I believe there is some better solution because this have big disadvantages:

• It works only on the initial deployment. If the pod with ingress-nginx is later moved to different zone (after restart), the connection is interrupted and the backend must be manually re-associated with the correct neg.
• We need to apply it in our environment where we don't use gcloud commands but everything is set via github actions with terraform.

After initial deployment it works well: But after restarting ingress-nginx pod it moves to different zone and backend stay attached to old one:

Our config is described in following tutorial:

https://github.com/robinpecha/gcp-regionalgke-httplb-negs-ingressnginx/blob/main/lb-negs-nging-reg.sh.md

---

GCP - HTTP LOAD BALANCER > NEGS > REGIONAL GKE CLUSTER > INGRESS-NGINX

Based on tutorial of Gabriel Hodoroaga .

Vars

Replace at least YOURDOMAIN.

CLUSTER_NAME="lb-negs-nging-reg"
REGION="europe-west2"
YOURDOMAIN="put-your-domain.here"
echo $CLUSTER_NAME ; echo $REGION ; echo $YOURDOMAIN

Create the cluster

gcloud container clusters create $CLUSTER_NAME --region $REGION --machine-type "e2-medium" --enable-ip-alias --num-nodes=2 

add the helm ingress-nginx

helm repo update
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx

Install the ingress-nginx

Create a file values.regional.yaml for ingress-nginx:

cat << EOF > values.regional.yaml
controller:
  service:
    type: ClusterIP
    annotations:
      cloud.google.com/neg: '{"exposed_ports": {"80":{"name": "ingress-nginx-80-neg"}}}'
EOF

And install it:

helm install -f values.regional.yaml ingress-nginx ingress-nginx/ingress-nginx

install dummy web server

Prepare config:

cat << EOF > dummy-app-lightweb.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: lightweb
spec:
  selector:
    matchLabels:
      app: dummy
  replicas: 3
  template:
    metadata:
      labels:
        app: dummy
    spec:
      containers:
      - name: lightweb
        image: alastairhm/alpine-lighttpd-php
        ports:
        - name: http
          containerPort: 80
        lifecycle:
          postStart:
            exec:
              command: ["/bin/sh", "-c", 'wget https://raw.githubusercontent.com/robinpecha/hello-world/main/php-header/index.php -P /var/www/']
---
apiVersion: v1
kind: Service
metadata:
  name: dummy-service
spec:
  type: NodePort
  ports:
  - port: 80
    targetPort: 80
  selector:
    app: dummy
EOF

Apply this config:

kubectl apply -f dummy-app-lightweb.yaml 

Now you can check if is your dummy web server works:

kubectl get pods
#  NAME                                        READY   STATUS    RESTARTS   AGE
#  ingress-nginx-controller-???????????-????   1/1     Running   0          5m8s
#  lightweb-???????????-????                   1/1     Running   0          4m35s
#  lightweb-???????????-????                   1/1     Running   0          4m35s
#  lightweb-???????????-????                   1/1     Running   0          4m35s

kubectl port-forward lightweb-???????????-???? 8080:80
#  Forwarding from 127.0.0.1:8080 -> 80
#  Forwarding from [::1]:8080 -> 80

Check in your browser http://localhost:8080

Ctrl+C

Create the ingress object

Prepare config. Dont forget to point dns record of $YOURDOMAIN to ip shown on end of this tutorial. Or simply edit your local hosts file for fake domain:

cat << EOF > dummy-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: dummy-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: "$YOURDOMAIN"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: dummy-service
            port:
              number: 80
EOF

And apply it:

kubectl apply -f dummy-ingress.yaml 

Find the network tags and zone of ingress

NETWORK_TAGS=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(tags.items[0])") ; echo $NETWORK_TAGS 

NODEZONE=$(gcloud compute instances list --filter="name=( $(kubectl get pod -l app.kubernetes.io/name=ingress-nginx -o jsonpath='{.items[0].spec.nodeName}') )" --format="value(zone)"); echo $NODEZONE 

Configure the firewall

gcloud compute firewall-rules create $CLUSTER_NAME-lb-fw --allow tcp:80 --source-ranges 130.211.0.0/22,35.191.0.0/16 --target-tags $NETWORK_TAGS 

Add health check configuration

gcloud compute health-checks create http app-service-80-health-check --request-path /healthz --port 80 --check-interval 60 --unhealthy-threshold 3 --healthy-threshold 1 --timeout 5 

Add the backend service

gcloud compute backend-services create $CLUSTER_NAME-lb-backend --health-checks app-service-80-health-check --port-name http --global --enable-cdn --connection-draining-timeout 300 

Attach our NEG to the backend service

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=$NODEZONE --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global 

Setup the frontend

gcloud compute url-maps create $CLUSTER_NAME-url-map --default-service $CLUSTER_NAME-lb-backend 
gcloud compute target-http-proxies create $CLUSTER_NAME-http-proxy --url-map $CLUSTER_NAME-url-map 
gcloud compute forwarding-rules create $CLUSTER_NAME-forwarding-rule --global --ports 80 --target-http-proxy $CLUSTER_NAME-http-proxy 

enable logging

gcloud compute backend-services update $CLUSTER_NAME-lb-backend --enable-logging --global 

Test

Give it some time to deploy...

IP_ADDRESS=$(gcloud compute forwarding-rules describe $CLUSTER_NAME-forwarding-rule --global --format="value(IPAddress)") ; echo $IP_ADDRESS
curl -s -I http://$IP_ADDRESS/ #404
echo curl -s -I http://$YOURDOMAIN/ #200

cleanup

# delete the forwarding-rule aka frontend
gcloud -q compute forwarding-rules delete $CLUSTER_NAME-forwarding-rule --global 
# delete the http proxy
gcloud -q compute target-http-proxies delete $CLUSTER_NAME-http-proxy 
# delete the url map
gcloud -q compute url-maps delete $CLUSTER_NAME-url-map 
# delete the backend
gcloud -q compute backend-services delete $CLUSTER_NAME-lb-backend --global
# delete the health check
gcloud -q compute health-checks delete app-service-80-health-check
# delete the firewall rule
gcloud -q compute firewall-rules delete $CLUSTER_NAME-lb-fw 

kubectl delete -f dummy-ingress.yaml 
kubectl delete -f dummy-app-lightweb.yaml 
helm delete ingress-nginx 

# delete the cluster
gcloud -q container clusters delete $CLUSTER_NAME --zone=$ZONE
# delete the NEG  
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-a
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-b
gcloud -q compute network-endpoint-groups delete ingress-nginx-80-neg --zone=$REGION-c
gcloud -q compute network-endpoint-groups list

Answer 1

Strange, it started working by simply adding the backend to all negs, even where there is no ingress-nginx.

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-a --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-b --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global

gcloud compute backend-services add-backend $CLUSTER_NAME-lb-backend --network-endpoint-group=ingress-nginx-80-neg --network-endpoint-group-zone=europe-west2-c --balancing-mode=RATE --capacity-scaler=1.0 --max-rate-per-endpoint=1.0 --global