带有 NFS 持久卷的 Kubernetes statefulset

Question

I have a kubernetes cluster and I have a simple deployment for mongodb with NFS persistent volume set. It works fine, but since resources like databases are stateful I thought of using Statefulset for the mongodb , but now the problem is, when I go through the documentation, statefulset has volumeClaimTemplates instead of volumes (in deployments).

But now the problem comes.

in a deployment do it like this:

PersistentVolume -> PersistentVolumeClaim -> Deployment

But how can we do this in Statefulset ?

Is it like:

volumeClaimTemplates -> StatefulSet

How can I set a PersistentVolume for the volumeClaimTemplates . If we don't use PersistentVolume for StatefulSet , how does it create he volume and WHERE does it create the volumes? Is in host machines (ie kubernetes worker nodes)?

Because I have a separate NFS provisioner I am using for the mongodb deployment (with replicasset=1), how can I use the same setup with StatefulSet ?

Here's the my mongo-deployment.yaml -> which I am going to transform into a statefulset as shown in the second code snippet ( mongo-stateful.yaml )

1. mongo-deployment.yaml

<omitted>
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs # name can be anything
spec:
  storageClassName: manual # same storage class as pvc
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany #  must be the same as PersistentVolume
  resources:
    requests:
      storage: 1Gi          
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment    
  labels:
    name: mongodb
spec:
  selector:
    matchLabels:
      app: mongodb
  replicas: 1
  template:
    metadata:
      labels: 
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        -  containerPort: 27017
        ... # omitted some parts for easy reading
        volumeMounts:
        - name: data  
          mountPath: /data/db
      volumes: 
        - name: data
          persistentVolumeClaim: 
            claimName: task-pv-claim    

2. mongo-stateful.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs # name can be anything
spec:
  storageClassName: manual # same storage class as pvc
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb-statefulset
spec:
  selector:
    matchLabels:
      name: mongodb-statefulset
  serviceName: mongodb-statefulset
  replicas: 2
  template:
    metadata:
      labels:
        name: mongodb-statefulset
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongodb
        image: mongo:3.6.4
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: db-data
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: db-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "manual"
      resources:
        requests:
          storage: 2Gi

But this is not working ( mongo-stateful.yaml ) pods are in pending state as when I describe it shows:

default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 pod has unbound immediate PersistentVolumeClaims

PS: Deployment works fine without any errors, problem is with Statefulset

Can someone please help me, how to write a statefulset with volumes?

Answer 1

If your storage class does not support dynamic volume provisionning, you have to manually create PVs and associated PVCs , using yaml files, then the volumeClaimTemplates will allow to link existing PVCs with your statefulset's pods.

Here is a working example: https://github.com/k8s-school/k8s-school/blob/master/examples/MONGODB-install.sh

You should:

• run it locally on https://kind.sigs.k8s.io/ , which support dynamic volume provisionning, so here PVCs and PVs will be created automatically
• export PVs and PVCs yaml files
• use these yaml file as template to create your PVs and PVCs for your NFS backend.

Here is what you will get on Kind:

$ ./MONGODB-install.sh               
+ kubectl apply -f 13-12-mongo-configmap.yaml
configmap/mongo-init created
+ kubectl apply -f 13-11-mongo-service.yaml
service/mongo created
+ kubectl apply -f 13-14-mongo-pvc.yaml
statefulset.apps/mongo created
$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          8m38s
mongo-1   2/2     Running   0          5m58s
mongo-2   2/2     Running   0          5m45s
$ kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
database-mongo-0   Bound    pvc-05247511-096e-4af5-8944-17e0d8222512   1Gi        RWO            standard       8m42s
database-mongo-1   Bound    pvc-f53c35a4-6fc0-4b18-b5fc-d7646815c0dd   1Gi        RWO            standard       6m2s
database-mongo-2   Bound    pvc-2a711892-eeee-4481-94b7-6b46bf5b76a7   1Gi        RWO            standard       5m49s
$ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
pvc-05247511-096e-4af5-8944-17e0d8222512   1Gi        RWO            Delete           Bound    default/database-mongo-0   standard                8m40s
pvc-2a711892-eeee-4481-94b7-6b46bf5b76a7   1Gi        RWO            Delete           Bound    default/database-mongo-2   standard                5m47s
pvc-f53c35a4-6fc0-4b18-b5fc-d7646815c0dd   1Gi        RWO            Delete           Bound    default/database-mongo-1   standard                6m1s

And a dump of a PVC (generated here by volumeClaimTemplate because odf kind dynamic volume provisionning):

$ kubectl get pvc database-mongo-0 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
    volume.kubernetes.io/selected-node: kind-worker2
  creationTimestamp: "2020-10-16T15:05:20Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: mongo
  managedFields:
    ...
  name: database-mongo-0
  namespace: default
  resourceVersion: "2259"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/database-mongo-0
  uid: 05247511-096e-4af5-8944-17e0d8222512
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-05247511-096e-4af5-8944-17e0d8222512
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

And the related PV:

kubectl get pv pvc-05247511-096e-4af5-8944-17e0d8222512 -o yaml     
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rancher.io/local-path
  creationTimestamp: "2020-10-16T15:05:23Z"
  finalizers:
  - kubernetes.io/pv-protection
  managedFields:
    ...
  name: pvc-05247511-096e-4af5-8944-17e0d8222512
  resourceVersion: "2256"
  selfLink: /api/v1/persistentvolumes/pvc-05247511-096e-4af5-8944-17e0d8222512
  uid: 3d1e894e-0924-411a-8378-338e48ba4a28
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: database-mongo-0
    namespace: default
    resourceVersion: "2238"
    uid: 05247511-096e-4af5-8944-17e0d8222512
  hostPath:
    path: /var/local-path-provisioner/pvc-05247511-096e-4af5-8944-17e0d8222512_default_database-mongo-0
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kind-worker2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard
  volumeMode: Filesystem
status:
  phase: Bound