在 kubernetes 中使用持久卷重新分析

Question

I have the following .yaml file to install redisinsights in kubernetes, with persistence support.

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: redisinsight-storage-class
provisioner: 'kubernetes.io/gce-pd'
parameters:
  type: 'pd-standard'
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: redisinsight-volume-claim
spec:
  storageClassName: redisinsight-storage-class
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 10Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: redisinsight #deployment name
  labels:
    app: redisinsight #deployment label
spec:
  replicas: 1 #a single replica pod
  selector:
    matchLabels:
      app: redisinsight #which pods is the deployment managing, as defined by the pod template
  template: #pod template
    metadata:
      labels:
        app: redisinsight #label for pod/s
    spec:
      initContainers:
        - name: change-data-dir-ownership
          image: alpine:3.6
          command:
            - chmod
            - -R
            - '777'
            - /db
          volumeMounts:
            - name: redisinsight
              mountPath: /db
      containers:
        - name: redisinsight #Container name (DNS_LABEL, unique)
          image: redislabs/redisinsight:1.6.1 #repo/image
          imagePullPolicy: Always #Always pull image
          volumeMounts:
            - name: redisinsight #Pod volumes to mount into the container's filesystem. Cannot be updated.
              mountPath: /db
          ports:
            - containerPort: 8001 #exposed conainer port and protocol
              protocol: TCP
      volumes:
        - name: redisinsight
          persistentVolumeClaim:
            claimName: redisinsight-volume-claim
---
apiVersion: v1
kind: Service
metadata:
  name: redisinsight
spec:
  ports:
    - port: 8001
      name: redisinsight
  type: LoadBalancer
  selector:
    app: redisinsight

However, it fails to launch and gives an error:

INFO 2020-07-03 06:30:08,117 redisinsight_startup Registered SIGTERM handler
ERROR 2020-07-03 06:30:08,131 redisinsight_startup Error in main()
Traceback (most recent call last):
  File "./startup.py", line 477, in main
ValueError: invalid literal for int() with base 10: 'tcp://10.69.9.111:8001'
Traceback (most recent call last):
  File "./startup.py", line 495, in <module>
  File "./startup.py", line 477, in main
ValueError: invalid literal for int() with base 10: 'tcp://10.69.9.111:8001'

But the same docker image, when run locally via docker as:

docker run -v redisinsight:/db -p 8001:8001 redislabs/redisinsight

works fine. What am I doing wrong?

It feels like redisinsights is trying to read port as an int but somehow gets a string and is confused. But I cannot understand how this works fine the local docker run.

Answer 1

UPDATE:

RedisInsight's kubernetes documentation has been updated recently. It clearly describes how to create a RedisInsight k8s deployment with and without a service.

IT also explains what to do when there's a service named "redisinsight" already:

Note - If the deployment will be exposed by a service whose name is 'redisinsight', set REDISINSIGHT_HOST and REDISINSIGHT_PORT environment variables to override the environment variables created by the service.

---

The problem is with the name of the service.

From thedocumentation , it is mentioned that RedisInsight has an environment variable REDISINSIGHT_PORT which can configure the port in which RedisInsight can run.

When you create a service in Kubernetes, all the pods that match the service, gets an environment variable <SERVICE_NAME>_PORT=<SERVICE_IP>:<SERVICE_PORT> .

So when you try to create the above mentioned service with name redisinsight , Kubernetes passes the service environment variable REDISINSIGHT_PORT=<SERVICE_IP>:SERVICE_PORT . But the port environment variable ( REDISINSIGHT_PORT ) is documented to be a port number and not an endpoint which makes the pod to crash when redisinsight running on the pod tries to use the environment variable as the port number.

So change the name of the service to be something different and not redisinsight and it should work.

Here's a quick deployment and service file:

Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: redisinsight #deployment name
  labels:
    app: redisinsight #deployment label
spec:
  replicas: 1 #a single replica pod
  selector:
    matchLabels:
      app: redisinsight #which pods is the deployment managing, as defined by the pod template
  template: #pod template
    metadata:
      labels:
        app: redisinsight #label for pod/s
    spec:
      containers:
      - name:  redisinsight #Container name (DNS_LABEL, unique)
        image: redislabs/redisinsight:1.6.3 #repo/image
        imagePullPolicy: IfNotPresent
        volumeMounts:
        - name: db #Pod volumes to mount into the container's filesystem. Cannot be updated.
          mountPath: /db
        ports:
        - containerPort: 8001 #exposed conainer port and protocol
          protocol: TCP
      volumes:
      - name: db
        emptyDir: {} # node-ephemeral volume https://kubernetes.io/docs/concepts/storage/volumes/#emptydir

Service:

apiVersion: v1
kind: Service
metadata:
  name: redisinsight-http # name should not be redisinsight 
spec:
  type: LoadBalancer
  ports:
    - port: 80
      targetPort: 8001
  selector:
    app: redisinsight

Please note the name of the service.

Logs of redisinsight pod:

 INFO 2020-09-02 11:46:20,689 redisinsight_startup Registered SIGTERM handler
 INFO 2020-09-02 11:46:20,689 redisinsight_startup Starting webserver...
 INFO 2020-09-02 11:46:20,689 redisinsight_startup Visit http://0.0.0.0:8001 in your web browser. Press CTRL-C to exit.

Also the service end point (from minikube):

$ minikube service list                                                                              
|----------------------|------------------------------------|--------------|-------------------------|
|      NAMESPACE       |                NAME                | TARGET PORT  |           URL           |
|----------------------|------------------------------------|--------------|-------------------------|
| default              | kubernetes                         | No node port |
| default              | redisinsight-http                  |           80 | http://172.17.0.2:30860 |
| kube-system          | ingress-nginx-controller-admission | No node port |
| kube-system          | kube-dns                           | No node port |
| kubernetes-dashboard | dashboard-metrics-scraper          | No node port |
| kubernetes-dashboard | kubernetes-dashboard               | No node port |
|----------------------|------------------------------------|--------------|-------------------------|

BTW, If you don't want to create a service at all (which is not related the question), you can do port forwarding:

kubectl port-forward <redisinsight-pod-name> 8001:8001

Answer 2

Problem is related to service, as it's interfering with the pod causing it to crash.

As we can read in the Redis docs Installing RedisInsight on Kubernetes

1. Once the deployment has been successfully applied and the deployment complete, access RedisInsight. This can be accomplished by exposing the deployment as a K8s Service or by using port forwarding, as in the example below:

kubectl port-forward deployment/redisinsight 8001

Open your browser and point to http://localhost:8001

Or a service which in your case while using GCP can look like this:

apiVersion: v1
kind: Service
metadata:
  name: redisinsight
spec:
  ports:
    - protocol: TCP
      port: 8001
      targetPort: 8001
      name: redisinsight
  type: LoadBalancer
  selector:
    app: redisinsight

Once the service receives the External-IP you can use it to access Redis.

crou@cloudshell:~ $ kubectl get service
NAME           TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)          AGE
kubernetes     ClusterIP      10.8.0.1     <none>          443/TCP          9d
redisinsight   LoadBalancer   10.8.7.0     34.67.171.112   8001:31456/TCP   92s

via http://34.67.171.112:8001/ in my example.

Answer 3

It happens to me too. In case anyone miss the conversation in the comments, here is the solution.

1. Deploy the redisinsight pod first and wait until it run successfully.

2. Deploy the service.

I think this is a bug and it is not really working because a pod can die anytime. It is kinda against the reason of using Kubernetes.

Someone have reported this issue here https://forum.redislabs.com/t/redisinsight-fails-to-launch-in-kubernetes/652/2

Answer 4

There are several problems with redisinsight running in k8s as suggested by the current documentation. I will list them below:

1. Suggestion is to use emptyDir Issue: Emptydir will most likely run out of space for larger redis clusters Solution: Use persistent volume
2. redisinsight docker container uses a redisinsight use Issue: redisinsight users does not ties to a specific uid. For this reason the persistent volume permissions cannot be set in a way that allows access to a pvc Solution: use cryptexlabs/redisinsight:latest which extends redislabs/redisinsight:latest but set uid for redisinsight to 777
3. default permissions do not allow access for redisinsight Issue: redisinsight will not be able to access the /db directory Solution: Use init container to set the directory permissions so that user 777 owns the /db directory
4. Suggestion is to use a nodeport for service Issue: this is a security hole Solution: Use ClusterIP instead and then use kubectl portforwarding to gain access or other secure access to redisinsight
5. Accessing rdb files locally is impractical. Problem: rdb files for large clusters must be downloaded and uploaded via the kubectl Solution: Use the s3 solution. If you are using kube2iam in an EKS cluster you'll need to create a special role that has access the bucket. Before that you must create a backup of your cluster and then export the backup following these instructions: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/backups-exporting.html

Summary Redisinsight is a good tool. But currently running it insight kubernetes cluster is an absolute nightmare and I t