调试 nfs 卷“无法为 pod 附加或安装卷”

Question

I've set up an nfs server that serves a RMW pv according to the example at https://github.com/kube.netes/examples/tree/master/staging/volumes/nfs

This setup works fine for me in lots of production environments, but in some specific GKE cluster instance, mount stopped working after pods restarted.

From kubelet logs I see the following repeating many times

Unable to attach or mount volumes for pod "api-bf5869665-zpj4c_default(521b43c8-319f-425f-aaa7-e05c08282e8e)": unmounted volumes=[shared-mount], unattached volumes=[geekadm.net deployment-role-token-6tg9p shared-mount]: timed out waiting for the condition; skipping pod

Error syncing pod 521b43c8-319f-425f-aaa7-e05c08282e8e ("api-bf5869665-zpj4c_default(521b43c8-319f-425f-aaa7-e05c08282e8e)"), skipping: unmounted volumes=[shared-mount], unattached volumes=[geekadm.net deployment-role-token-6tg9p shared-mount]: timed out waiting for the condition

Manually mounting the nfs on any of the nodes work just fine: mount -t nfs <service ip>:/ /tmp/mnt

How can I further debug the issue? Are there any other logs I could look at besides kubelet?

Answer 1

In case the pod gets kicked out of the node because the mount is too slow, you may see messages like that in logs.

Kubelets even inform about this issue in logs.
Sample log from Kubelets:
Setting volume ownership for /var/lib/kubelet/pods/c9987636-acbe-4653-8b8d- aa80fe423597/volumes/kube.netes.io~gce-pd/pvc-fbae0402-b8c7-4bc8-b375- 1060487d730d and fsGroup set. If the volume has a lot of files then setting volume ownership could be slow, see https://github.com/kube.netes/kube.netes/issues/69699

Cause:
The pod.spec.securityContext.fsGroup setting causes kubelet to run chown and chmod on all the files in the volumes mounted for given pod. This can be a very time consuming thing to do in case of big volumes with many files.

By default, Kube.netes recursively changes ownership and permissions for the contents of each volume to match the fsGroup specified in a Pod's securityContext when that volume is mounted. From the document .

Solution:
You can deal with it in the following ways.

1. Reduce the number of files in the volume.
2. Stop using the fsGroup setting.

Answer 2

Did you specify an nfs version when mounting command-line? I had the same issue on AKS, but inspired by https://stackoverflow.com/a/71789693/1382108 I checked the nfs versions. Noticed my PV had a vers=3. When I tried mounting command-line using mount -t nfs -o vers=3 command just hung, with vers=4.1 it worked immediately. Changed the version in my PV and next Pod worked just fine.