Kubernetes - Unable to connect to Redis Pods from inside Kubernetes Cluster

Question

I am trying to connect to some redis pods in my kubernetes cluster but I cannot make it work. I call the redis service both by trying to use the service name as my hostname in the program connecting to the redis cluster redis-sentinel:26379 or with the direct list of endpoints from my 3 pods running the redis image 10.0.10.xxx:26379 . I know the redis cluster works as I can run my program outside of docker/kubernetes altogether and turn the redis-sentinel service into a NodePort and it can connect and run just fine. But I am unable to connect to this redis cluster from other pods inside of kubernetes.

Using Docker Desktop as my Kubernetes environment.

Redis pod Service:

kind: Service
metadata:
  labels:
    name: sentinel
    role: service
  name: redis-sentinel
spec:
  ports:
    - port: 26379
      targetPort: 26379
  selector:
    redis-sentinel: "true"

Redis-Sentinel ReplicaController:

kind: ReplicationController
metadata:
  name: redis-sentinel
spec:
  replicas: 3
  selector:
    redis-sentinel: "true"
  template:
    metadata:
      labels:
        name: redis-sentinel
        redis-sentinel: "true"
        role: sentinel
    spec:
      containers:
      - name: sentinel
        image: k8s.gcr.io/redis:v1
        env:
          - name: SENTINEL
            value: "true"
        ports:
          - containerPort: 26379

Redis Master ReplicaController:

kind: ReplicationController
metadata:
  name: redis
spec:
  replicas: 3
  selector:
    name: redis
  template:
    metadata:
      labels:
        name: redis
        role: master
    spec:
      containers:
      - name: redis
        image: k8s.gcr.io/redis:v1
        ports:
        - containerPort: 6379
        resources:
          limits:
            cpu: "0.1"
        volumeMounts:
        - mountPath: /redis-master-data
          name: data
      volumes:
        - name: data

@Matt, trying to PING redis-sentinel timesout and nc redis-sentinel 26379 seems to do nothing.

@FrankYuchengGu yes the DNS service is running, but seems to fail. Running a busybox image returns this when running nslookup redis-sentinel

** server can't find redis-sentinel.default.svc.cluster.local: NXDOMAIN 

*** Can't find redis-sentinel.svc.cluster.local: No answer 
*** Can't find redis-sentinel.cluster.local: No answer 
*** Can't find redis-sentinel.default.svc.cluster.local: No answer 
*** Can't find redis-sentinel.svc.cluster.local: No answer 
*** Can't find redis-sentinel.cluster.local: No answer

It looks as though the busybox image was having issues with the nslookup command. using the dnsutils image from Kubernetes DNS Debugging page can find the service

$ kubectl exec -ti dnsutils -- nslookup redis-sentinel
Server:         10.96.0.10
Address:        10.96.0.10#53

Name:   redis-sentinel.default.svc.cluster.local
Address: 10.110.45.31

However, it seems that trying either redis-sentinel.default.svc.cluster.local:26379 or 10.110.45.31:26379 does not work either and my program still is unable to find the redis cluster.

Answer 1

I guess you used this tutorial on Github. When I deployed YAMLs you have provided i encounterd many issues that I couldn't connect. In mentioned tutorial you can find information that at first you must create Master Pod .

We will use the shared network namespace to bootstrap our Redis cluster. In particular, the very first sentinel needs to know how to find the master (subsequent sentinels just ask the first sentinel). Because all containers in a Pod share a network namespace, the sentinel can simply look at $(hostname -i):6379.

Then Service and later redis server . After everything set you just delete this pod. Fast Steps to deploy can be found here

$ kubectl get svc,pods -o wide
NAME                     TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE   SELECTOR
service/kubernetes       ClusterIP   10.97.0.1      <none>        443/TCP     21h   <none>
service/redis-sentinel   ClusterIP   10.97.13.152   <none>        26379/TCP   71m   redis-sentinel=true

NAME                       READY   STATUS    RESTARTS   AGE   IP          NODE                                   NOMINATED NODE   READINESS GATES
pod/redis-6kb5p            1/1     Running   0          71m   10.32.2.7   gke-redis-default-pool-bc40bcaa-08sj   <none>           <none>
pod/redis-sentinel-qf9l8   1/1     Running   0          71m   10.32.1.5   gke-redis-default-pool-bc40bcaa-txmt   <none>           <none>
pod/redis-sentinel-rnsw6   1/1     Running   0          71m   10.32.2.8   gke-redis-default-pool-bc40bcaa-08sj   <none>           <none>
pod/redis-sentinel-sbn8f   1/1     Running   0          71m   10.32.1.7   gke-redis-default-pool-bc40bcaa-txmt   <none>           <none>
pod/redis-sq2g2            1/1     Running   0          71m   10.32.1.6   gke-redis-default-pool-bc40bcaa-txmt   <none>           <none>
pod/redis-wv5q2            1/1     Running   0          71m   10.32.1.8   gke-redis-default-pool-bc40bcaa-txmt   <none>           <none>

$ kubectl get ep
NAME             ENDPOINTS                                         AGE
redis-sentinel   10.32.1.5:26379,10.32.1.7:26379,10.32.2.8:26379   72m

In addition, as you can also check this tutorial using Redis 3.x.

How to Connect

To running Redis container , execute command

kubectl exec -ti <pod-name> -- redis-cli 

$ kubectl exec -ti redis-6kb5p redis-cli
127.0.0.1:6379> info
# Server
redis_version:2.8.19
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:f308ca06a4d63700

To connect to redis using service , you should enter the pod.

$ kubectl exec -ti redis-sentinel-qf9l8 /bin/bash

If you will check env inside this container, you will be able to find some kubernetes configuration.

root@redis-sentinel-qf9l8:/data# env
HOSTNAME=redis-sentinel-qf9l8
REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-2.8.19.tar.gz
KUBERNETES_PORT_443_TCP_PORT=443
KUBERNETES_PORT=tcp://10.97.0.1:443
TERM=xterm
KUBERNETES_SERVICE_PORT=443
KUBERNETES_SERVICE_HOST=10.97.0.1
REDIS_SENTINEL_PORT_26379_TCP_ADDR=10.97.13.152
REDIS_SENTINEL_SERVICE_HOST=10.97.13.152

Now based on script from here

master=$(redis-cli -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT}

You can connect to service using Kubernetes envs.

root@redis-sentinel-qf9l8:/data# redis-cli -h ${REDIS_SENTINEL_SERVICE_HOST} -p ${REDIS_SENTINEL_SERVICE_PORT}
10.97.13.152:26379>

Or using specific information $ redis-cli -h <service-name> -p <service-port>

root@redis-sentinel-qf9l8:/data# redis-cli -h redis-sentinel -p 26379
redis-sentinel:26379>

As additional information.

You cannot ping service with ClusterIP as it's virtual address managed by kube-proxy daemon. More details can be found in official docs .