Azure Kubernetes 上的 Redis 分片集群

Question

I have followed these links

1. https://medium.com/zero-to/setup-persistence-redis-cluster-in-kubertenes-7d5b7ffdbd98 ,
2. https://github.com/sanderploegsma/redis-cluster

to build a Redis sharded cluster on AKS, But they are connected by using Pod IP's but I need to connect to that cluster using "Python" to feed the data into it, Since they are connected internally using Pod IPs I am not able to connect using Python. Alternatively instead of 6 replicas of one statefulset, I have created 6 different statefulsets with 6 services and all exposed externally as "Load Balancer", But this command they used to make a cluster

kubectl exec -it redis-cluster-0 -- redis-cli --cluster create --cluster-replicas 1 \
$(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')

I don't know how to edit it to use loadbalancer IP instead of Pod IP, so with this 6 external IP's I am not able to create a cluster. I need a Redis sharded cluster on Azure Kubernetes Services which can be accessible externally from Python library "redis-py-cluster"

Thanks in advance

Answer 1

theory

It is not very clear from your question as there is no answer formy comment under the initial post, but it looks like you have been following this guide: Setup Persistence Redis Cluster in Kubernetes .

Lets decompose the command you are referring to:

kubectl exec -it redis-cluster-0 -- redis-trib create --replicas 1 $(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')

the kubectl syntax is:

kubectl exec (POD | TYPE/NAME) [-c CONTAINER] [flags] -- COMMAND [args...] [options]

That is why: kubectl exec -it redis-cluster-0 stands for "Execute a command in a container" with "Pass stdin to the container" and "Stdin is a TTY" options on redis-cluster-0 pod.

The second part is redis-trib create --replicas 1 <IPs> and

$(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')

merely lists IPs of pods in your StatefulSet/s created with redis-cluster-deployment.yaml .

Taking into consideration that the app=redis-cluster label we are using in that command is taken from the spec/template/metadata/labels you can adjust the command the way you like or just list the IPs manually insted of getting them with kubectl get pods -l

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
...
spec:
  serviceName: redis-cluster
  replicas: 6
  template:
    metadata:
      labels:
        app: redis-cluster    # This is the label we are using in that command

my answer :

You can either use the IPs or adjust labels you are using in kubectl get pods -l ... part of the command. Can't provide precise answer as I didn't receive steps to reproduce.

my attempt :

I'm not sure how exactly you've been creating your statefulsets; however, you still can create a redis cluster on them.

In my case I have used the following YAMLs: redis-cluster-deployment-1.yaml and redis-cluster-deployment-2.yaml .

$ kubectl create -f redis-cluster-deployment-1.yaml 
statefulset.apps/redis-cluster-set-1 created

$ kubectl create -f redis-cluster-deployment-2.yaml 
statefulset.apps/redis-cluster-set-2 created

$ kubectl get pods -o wide 
NAME                    READY   STATUS    RESTARTS   AGE   IP           NODE                                            
redis-cluster-set-1-0   1/1     Running   0          52m   10.12.0.22   gke-6v3n
redis-cluster-set-1-1   1/1     Running   0          51m   10.12.1.17   gke-m7z8
redis-cluster-set-1-2   1/1     Running   0          50m   10.12.1.18   gke-m7z8
redis-cluster-set-2-0   1/1     Running   0          51m   10.12.0.23   gke-6v3n
redis-cluster-set-2-1   1/1     Running   0          50m   10.12.1.19   gke-m7z8
redis-cluster-set-2-2   1/1     Running   0          14m   10.12.0.24   gke-6v3n


$ kubectl exec -it redis-cluster-set-1-0 -- redis-trib create --replicas 1 $(kubectl get pods -l app=redis-cluster-set-app -o jsonpath='{range.items[*]}{.status.podIP}:6379 ')
>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
10.12.0.22:6379
10.12.1.17:6379
10.12.1.18:6379
Adding replica 10.12.1.19:6379 to 10.12.0.22:6379
Adding replica 10.12.0.24:6379 to 10.12.1.17:6379
Adding replica 10.12.0.23:6379 to 10.12.1.18:6379
....

the important part here is to specify correct pod name and app= label .

In my case , the $(kubectl get pods -l app=redis-cluster-set-app -o jsonpath='{range.items[*]}{.status.podIP}:6379 ') command results in a following list of IPs (you can compare them with the output from above.

$ kubectl get pods -l app=redis-cluster-set-app -o jsonpath='{range.items[*]}{.status.podIP}:6379 '
10.12.0.22:6379 10.12.1.17:6379 10.12.1.18:6379 10.12.0.23:6379 10.12.1.19:6379 10.12.0.24:6379

Hope that helps.

Update (06-Dec-2019):

have created a service LoadBalancer" and I am pretty sure I have given a correct "serviceName" under "spec". That LoadBalancer gave me an external Ip. After I created a redis sharded cluster, when I try to connect to that external IP using python, It was able to connect to only one node of redis and asks for other 5 nodes and eventually failed with "Timeout" error

I did the same and tested it Redis cluster on 3 nodes:

$ kubectl get pods -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP           
redis-cluster-set-1-0   1/1     Running   0          20h   10.12.1.30
redis-cluster-set-1-1   1/1     Running   0          20h   10.12.0.29
redis-cluster-set-1-2   1/1     Running   0          20h   10.12.1.31

Created some data and tried to get it out from Redis cluster.

$ kubectl exec -it redis-cluster-set-1-0 -- redis-cli SET Australia Sydney
OK
$ kubectl exec -it redis-cluster-set-1-0 -- redis-cli GET Australia
"Sydney"

it worked so I tried to query the same data from another node in cluster

$ kubectl exec -it redis-cluster-set-1-2 -- redis-cli GET Australia
(error) MOVED 1738 10.12.1.30:6379

Redis cluster replied with Internal IP in MOVED response.

I think that internal IP adress is the reason for that "timeOut" error.

At the same time it is possible to get data when logged into that pod directly

$ kubectl exec -it redis-cluster-set-1-2 -- bash
root@redis-cluster-set-1-2:/data# redis-cli GET Australia
(error) MOVED 1738 10.12.1.30:6379

root@redis-cluster-set-1-2:/data# redis-cli -h 10.12.1.30  GET Australia
"Sydney"

From this Github Issue it seems that redis doesn't proxy the connection through the Redis cluster, to correct node but rather simply passes the IP address of the appropriate server back to the client, which then initiates the connection directly.

(slight off-topic, but still worth mentioning). While checking on that I have found an [another document] that explains ( https://cloud.google.com/memorystore/docs/redis/connect-redis-instance-gke ) how to connect to a Redis instance from a Google Kubernetes Engine cluster (with LoadBalancer).