Kubernetes NFS server pod mount works with pod ip but not with kubernetes service

Question

I created a nfs server in a pod to use it as a volume. When creating another pod with a volume, the volume mount does work with the ip of the nfs pod. Since this ip is not guaranteed to stay the same, I added a service for my nfs pod and added a fixed cluster ip. When starting the container with the volume mount, it always fails with the following error:

Unable to mount volumes for pod "nginx_default(35ecd8ec-a077-11e8-b7bc-0cc47a9aec96)": timeout expired waiting for volumes to attach or mount for pod "default"/"nginx". list of unmounted volumes=[nfs-demo]. list of unattached volumes=[nfs-demo nginx-test-account-token-2dpgg]

    apiVersion: v1
    kind: Pod
    metadata:
      name: nfs-server
      labels:
        name: nfs-server
    spec:
      containers:
      - name: nfs-server
        image: my-nfs-server:v1
        args: ["/exports"]
        securityContext:
          privileged: true
    ---
    kind: Service
    apiVersion: v1
    metadata:
      name: nfs-service
    spec:
      selector:
        name: nfs-server
      clusterIP: "10.96.0.3"
      ports:
        - name: nfs
          port: 2049
          protocol: UDP
        - name: mountd
          port: 20048
          protocol: UDP   
        - name: rpcbind
          port: 111
          protocol: UDP
        - name: nfs-tcp
          port: 2049
          protocol: TCP
        - name: mountd-tcp
          port: 20048
          protocol: TCP
        - name: rpcbind-tcp
          port: 111
          protocol: TCP

My pod trying to mount the server:

    apiVersion: v1
    kind: Pod
    metadata:
      name: nginx
      labels:
        name: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        volumeMounts:
        - mountPath: "/exports"
          name: nfs-demo
        securityContext:
          privileged: true
      securityContext:
        supplementalGroups: [100003]
      serviceAccountName: nginx-test-account
      volumes:
      - name: nfs-demo
        nfs:
          server: 10.96.0.3
          path: "/exports"
          readOnly: false

I used this as a base for my nfs server image:

https://github.com/cpuguy83/docker-nfs-server

https://medium.com/@aronasorman/creating-an-nfs-server-within-kubernetes-e6d4d542bbb9

Does anyone have an idea why the mount ist working with the pod ip but not with the service ip?

Answer 1

I found a new way to solve this problem ,you can set nfs-server port to be fixed ,then mount nfs-server by service . you can refer to https://wiki.debian.org/SecuringNFS

Answer 2

Try removing the ClusterIP ip address (let kube assign an ip to nfs service) and use the name 'nfs-service' in your volume mount definition. Make sure that the nginx pod and the nfs service are on the same namespace.

Answer 3

As mentioned by Bal Chua you probably didn't export the nfs port in nfs-server pod definition.

nfs-server-pod.yaml

apiVersion: v1beta1
kind: Pod
id: nfs-server
desiredState:
  manifest:
    version: v1beta1
    id: nfs-server
    containers:
      - name: nfs-server
        image: jsafrane/nfs-data
        privileged: true
        ports:
          - name: nfs
            containerPort: 2049
            protocol: tcp
labels:
  name: nfs-server

nfs-server-service.yaml

id: nfs-server
kind: Service
apiVersion: v1beta1
port: 2049
protocol: tcp
selector:
  name: nfs-server

Taken from example of NFS volume page.

Answer 4

I found the solution to my problem:

There were ports missing in my service , not the pod. To find the ports I needed, I opened a console to my pod (kubectl exec) and used the " rpcinfo -p " command to list the ports needed for the service.

It does fix the connection problem, but only temporarily. These ports are not static, so it is not better than using the port IP itself. I do think it is possible to configure static ports though.

If anyone with a similar problem needs further reading:

http://tldp.org/HOWTO/NFS-HOWTO/security.html

https://wiki.debian.org/SecuringNFS

The second problem I encountered: the mount only worked if the nfs-server pod and the pod mounting it were on the same node. I could fix it when updating to kubernetes version 1.11.

Since my original problem is solved, I consider my question answered though.