Unable to mount local PersistentVolume in k8s 1.13

Question

I'm trying to deploy a stateful set with a persistent volume claim on a bare metal kubernetes cluster (v1.13) but the pod times out when trying to mount the volume.

I have a local-storage storage class defined:

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: local-storage
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer

I have a PV defined:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: cassandradev1
  labels:
    app: cassandra
    environment: dev
spec:
  storageClassName: local-storage
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteOnce
  local:
    path: "/data1/cassandradev1"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node1

And I have a stateful set that issues a claim (truncated):

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: cassandra-set
spec:
  ...
  volumeClaimTemplates:
  - metadata:
      name: cassandra-data
    spec:
      selector:
        matchLabels:
          app: "cassandra"
          environment: "dev"
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "local-storage"
      resources:
        requests:
          storage: 1Ti

When I try to apply the stateful set, the Pod gets scheduled but times out:

Normal   Scheduled    2m13s  default-scheduler            Successfully assigned default/cassandra-set-0 to my-node1
Warning  FailedMount  13s    kubelet, my-node1  Unable to mount volumes for pod "cassandra-set-0 (dd252f77-fda3-11e8-96d3-1866dab905dc)": timeout expired waiting for volumes to attach or mount for pod "default"/"cassandra-set-0". list of unmounted volumes=[cassandra-data]. list of unattached volumes=[cassandra-data default-token-t2dg8]

If I look at the logs for the controller I see an error message for no volume plugin matched:

kubectl logs pod/kube-controller-manager -n kube-system
W1212 00:51:24.218114       1 plugins.go:845] FindExpandablePluginBySpec(cassandradev1) -> err:no volume plugin matched

Any ideas on where to look next?

Answer 1

First of all PV definition is incorrect, there is no hostPath in local-storage class. This is how you should define your local storage PV:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: cassandradev1
  labels:
    app: cassandra
    environment: dev
spec:
  storageClassName: local-storage
  capacity:
    storage: 1Ti
  accessModes:
    - ReadWriteOnce
  local:
    path: "/data1/cassandradev1"
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - my-node1

Also keep in mind unlike hostPath, /data1/cassandradev1 should exist on my-node1 , local-storage doesn't automatically create the path and when you deploy statefulset and path is not there, it will give error related to mounting.

This should resolve your issue. Hope this helps.

EDIT: So, I setup the cassandra statefulset with local-storage by following yaml files. I have omitted some config maps and so it will not work as it is. Could you please try to check what is different there:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  generation: 1
  labels:
    state: cassandra
  name: cassandra
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: cassandra
  serviceName: cassandra
  template:
    metadata:
      annotations:
        pod.alpha.kubernetes.io/initialized: "true"
      creationTimestamp: null
      labels:
        app: cassandra
    spec:
      containers:
      - args:
        - chmod -R 777 /logs/; /on_start.sh
        command:
        - /bin/sh
        - -c
        image: <image>
        imagePullPolicy: Always
        name: cassandra
        ports:
        - containerPort: 9042
          protocol: TCP
        resources: {}
        volumeMounts:
        - mountPath: /data
          name: data
        imagePullSecrets:
      - name: gcr-imagepull-json-key
  volumeClaimTemplates:
  - metadata:
      creationTimestamp: null
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: local-storage

PV.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    type: local
  name: cassandra-data-vol-0
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 10Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: data-cassandra-0
    namespace: default
  local:
    path: /data/cassandra-0
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - ip-10-0-1-91.ec2.internal
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage

Make sure /data/cassandra-0 exists before you create PV. Let me know if you face any issues.