Redis Cluster Client doesn't work with Redis cluster on GKE

Question

My setup has a K8S Redis cluster with 8 nodes and 32 pods across them and a load balancer service on top.

I am using a Redis cluster client to access this cluster using the load balancer's external IP. However, when handling queries, as part of Redis cluster redirection (MOVED / ASK), the cluster client receives internal IP addresses of the 32 Pods, connection to which fails within the client.

For example, I provide the IP address of the load balancer (35.245.51.198:6379) but the Redis cluster client throws errors like - Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Failed connecting to host 10.32.7.2:6379, which is an internal Pod IP.

Any ideas about how to deal with this situation will be much appreciated.

Thanks in advance.

Answer 1

If you're running on GKE, you can NAT the Pod IP using the IP masquerade agent :

Using IP masquerading in your clusters can increase their security by preventing individual Pod IP addresses from being exposed to traffic outside link-local range (169.254.0.0/16) and additional arbitrary IP ranges

Your issue specifically is that, the pod range is on 10.0.0.0/8 , which is by default a non-masquerade CIDR .

You can change this using a ConfigMap to treat that range as masquerade so that it picks the node's external IP as source address.

Alternatively, you can change the pod range in your cluster to anything that is masked.

Answer 2

I have been struggling with the same problem in installing the bitnami/redis-cluster on gke.

In order to have the right networking settings you should create the cluster setting as a public cluster

The equivalent command line for creating the cluster in MYPROJECT is:

gcloud beta container --project "MYPROJECT" clusters create "redis-cluster" --zone "us-central1-c" --no-enable-basic-auth --cluster-version "1.21.5-gke.1802" --release-channel "regular" --machine-type "e2-medium" --image-type "COS_CONTAINERD" --disk-type "pd-standard" --disk-size "100" --metadata disable-legacy-endpoints=true --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" --num-nodes "3" --logging=SYSTEM,WORKLOAD --monitoring=SYSTEM --no-enable-ip-alias --network "projects/MYPROJECT/global/networks/default" --subnetwork "projects/oddsjam/regions/us-central1/subnetworks/default" --no-enable-intra-node-visibility --no-enable-master-authorized-networks --addons HorizontalPodAutoscaling,HttpLoadBalancing,GcePersistentDiskCsiDriver --enable-autoupgrade --enable-autorepair --max-surge-upgrade 1 --max-unavailable-upgrade 0 --workload-pool "myproject.svc.id.goog" --enable-shielded-nodes --node-locations "us-central1-c"

Then you need to create as many External IP addresses in the Network VPC product. Those IP addresses will be picked by the Redis nodes automatically.

Then you are ready to get the values.yaml of the Bitnami Redis Cluster Helm chart and change the conf accordingly to your use case. Add the list of external ips you created to the cluster.externalAccess.loadBalancerIP value.

Finally, you can run the command to install a Redis cluster on GKE by running

helm install cluster-name -f values.yaml bitnami/redis-cluster

This command will give you the password of the cluster. you can use redis-client to connect to the new cluster with:

redis-cli -c -h EXTERNAL_IP -p 6379 -a PASSWORD