Kubernetes：使用 NFS 的动态持久卷配置

Question

I have multi node kubernetes setup. I am trying to allocate a Persistent volume dynamically using storage classes with NFS volume plugin. I found storage classes examples for glusterfs, aws-ebs, etc.but, I didn't find any example for NFS. If I create PV and PVC only then NFS works very well(Without storage class). I tried to write storage class file for NFS, by referring other plugins. please refer it below,

nfs-storage-class.yaml

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  namespace: kube-system
  name: my-storage
  annotations:
    storageclass.beta.kubernetes.io/is-default-class: "true"
  labels:
    kubernetes.io/cluster-service: "true"

provisioner: kubernetes.io/nfs
parameters:
  path: /nfsfileshare
  server: <nfs-server-ip> 

nfs-pv-claim.yaml

apiVersion: v1
metadata:
  name: demo-claim
  annotations:
    volume.beta.kubernetes.io/storage-class: my-storage
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 3Gi

It didn't worked. So, my question is, Can we write a storage class for NFS? Does it support dynamic provisioing?

Answer 1

As of August 2020, here's how things look for NFS persistence on Kubernetes:

You can

• Put an NFS volume on a Pod directly:

apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: k8s.gcr.io/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /test-pd
      name: test-volume
  volumes:
  - name: test-volume
    nfs:
      path: /foo/bar
      server: wherever.dns

• Manually create a Persistent Volume backed by NFS, and mount it with a Persistent Volume Claim (PV spec shown below):

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv0003
spec:
  capacity:
    storage: 5Gi
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: slow
  mountOptions:
    - hard
    - nfsvers=4.1
  nfs:
    path: /tmp
    server: 172.17.0.2

• Use the (now deprecated) NFS PV provisioner from external-storage . This was last updated two years ago, and has been officially EOL'd, so good luck. With this route, you can make a Storage Class such as the one below to fulfill PVCs from your NFS server.
  • Update: There is a new incarnation of this provisioner as kubernetes-sigs/nfs-subdir-external-provisioner ! It seems to work in a similar way to the old nfs-client provisioner, but is much more up-to-date. Huzzah!

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: example-nfs
provisioner: example.com/nfs
mountOptions:
  - vers=4.1

• Evidently, CSI is the future, and there is a NFS CSI driver. However, it doesn't support dynamic provisioning yet, so it's not really terribly useful.
  • Update (December 2020): Dynamic provisioning is apparently in the works (on master , but not released) for the CSI driver.
• You might be able to replace external-storage 's NFS provisioner with something from the community, or something you write. In researching this problem, I stumbled on a provisioner written by someone on GitHub, for example. Whether such provisioners perform well, are secure, or work at all is beyond me, but they do exist.

Answer 2

I'm looking into doing the same thing. I found https://github.com/kubernetes-incubator/external-storage/tree/master/nfs , which I think you based your provisioner on?

I think an nfs provider would need to create a unique directory under the path defined. I'm not really sure how this could be done.

Maybe this is better of as an github issue on the kubernetes repo.

Answer 3

Dynamic storage provisioning using NFS doesn't work, better use glusterfs. There's a good tutorial with fixed to common problems while setting up. http://blog.lwolf.org/post/how-i-deployed-glusterfs-cluster-to-kubernetes/

Answer 4

I also tried to enable the NFS provisioner on my kubernetes cluster and at first it didn't work, because the quickstart guide does not mention that you need to apply the rbac.yaml as well (I opened a PR to fix this).

The nfs provisioner works fine for me if I follow these steps on my cluster: https://github.com/kubernetes-incubator/external-storage/tree/master/nfs#quickstart

$ kubectl create -f deploy/kubernetes/deployment.yaml
$ kubectl create -f deploy/kubernetes/rbac.yaml
$ kubectl create -f deploy/kubernetes/class.yaml

Then you should be able to create PVCs like this:

$ kubectl create -f deploy/kubernetes/claim.yaml

You might want to change the folders used for the volume mounts in deployment.yaml to match it with your cluster.

Answer 5

The purpose of StorageClass is to create storage, eg from cloud providers (or "Provisioner" as they call it in the kubernetes docs). In case of NFS you only want to get access to existing storage and there is no creation involved. Thus you don't need a StorageClass . Please refer to this blog .

Answer 6

You can directly create pv, pvc shown below. This is an example PV and PVC for private registry, it should work for most of the apps. Make sure to install nfs server and configure the mount entries in nfs server's /etc/exports for access.

example:
/mnt/disks/vol1 <IP ADDRESS OF THE NODE>(rw,sync,no_root_squash,no_all_squash)

registry-pv.yml -

apiVersion: v1
kind: PersistentVolume
metadata:
  name: registry-pv
  namespace: kube-system
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  nfs:
    path: /mnt/disks/vol1
    server: <IP-Address of NFS SERVER>
  persistentVolumeReclaimPolicy: Retain
  storageClassName: registry-storage

registry-pvc.yml -

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  accessModes:
    - ReadWriteMany
  storageClassName: registry-storage
  volumeName: registry-pv
  resources:
    requests:
      storage: 50Gi

create pvc, pv -

kubectl create -f registry-pv.yml 
kubectl create -f registry-pvc.yml

Once created you should see the pv bound to the pvc -

Answer 7

If you are using AWS, I believe you can use this image to create a NFS server:

https://hub.docker.com/r/alphayax/docker-volume-nfs