ELK Stateful - ERROR: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims

Question

I need to create a cluster with ELK. Elastic should be Stateful, I'm not able to attach disks; the error highlighted below occurs. Does anyone have any solution?

Sincreley, Pablo

Message error: 0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims.

YAML:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elasticsearch-logging
  namespace: elk-iot-cloud
  labels:
    k8s-app: elasticsearch-logging
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
rules:
- apiGroups:
  - ""
  resources:
  - "services"
  - "namespaces"
  - "endpoints"
  verbs:
  - "get"
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: elk-iot-cloud
  name: elasticsearch-logging
  labels:
    k8s-app: elasticsearch-logging
subjects:
- kind: ServiceAccount
  name: elasticsearch-logging
  namespace: elk-iot-cloud
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: elasticsearch-logging
  apiGroup: ""
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch-logging
  namespace: elk-iot-cloud
  labels:
    k8s-app: elasticsearch-logging
spec:
  serviceName: elasticsearch-logging
  replicas: 3
  updateStrategy:
    type: RollingUpdate
  selector:
    matchLabels:
      k8s-app: elasticsearch-logging
  template:
    metadata:
      labels:
        k8s-app: elasticsearch-logging
    spec:
      serviceAccountName: elasticsearch-logging
      containers:
      - image: docker.elastic.co/elasticsearch/elasticsearch:7.15.1    
        name: elasticsearch-logging
        ports:
        - containerPort: 9200
          name: db
          protocol: TCP
        - containerPort: 9300
          name: transport
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          # sets a list of master-eligible nodes in the cluster.
          - name: discovery.seed_hosts
            value: 'elasticsearch-logging-0.elasticsearch-logging.elk-iot-cloud.svc.cluster.local,elasticsearch-logging-1.elasticsearch-logging.elk-iot-cloud.svc.cluster.local ,elasticsearch-logging-2.elasticsearch-logging.elk-iot-cloud.svc.cluster.local'
          # specifies a list of master-eligible nodes that will participate in the master election process.
          - name: cluster.initial_master_nodes
            value: 'elasticsearch-logging-0,elasticsearch-logging-1,elasticsearch-logging-2'
          - name: ES_JAVA_OPTS
            value: '-Xms1g -Xmx1g'
          - name: ELASTICSEARCH_USERNAME
            value: 'elastic'
          - name: ELASTIC_PASSWORD
            value: 'elastic'
          #- name: xpack.license.self_generated.type
          #  value: "basic"
          - name: xpack.security.enabled
            value: 'true'
          #- name: xpack.security.transport.ssl.enabled
          #  value: 'true'
          #- name: xpack.security.audit.enabled
          #  value: 'true'
          - name: xpack.monitoring.collection.enabled
            value: 'true'
      volumes:
      - name: elasticsearch-logging
        emptyDir: {}
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data      
      - name: elasticsearch-logging-init
        image: busybox
        command: ["/sbin/sysctl", "-w", "vm.max_map_count=262144"]        
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch-logging
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: do-block-storage
      resources:
        requests:
          storage: 50Gi
---
apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: elk-iot-cloud
  labels:
    k8s-app: elasticsearch-logging
spec:
  ports:
  - port: 9200
    protocol: TCP
    targetPort: db
  selector:
    k8s-app: elasticsearch-logging
---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: data
  namespace: elk-iot-cloud
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-pv0
  namespace: elk-iot-cloud
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: data
  local: 
    path: /mnt/disk/vol0
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - elasticsearch-logging-0
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-pv1
  namespace: elk-iot-cloud
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: data
  local: 
    path: /mnt/disk/vol1 
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - elasticsearch-logging-1           
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: data-pv2
  namespace: elk-iot-cloud
spec:
  capacity:
    storage: 5Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: data
  local: 
    path: /mnt/disk/vol2
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/hostname
            operator: In
            values:
              - elasticsearch-logging-2     

Config: Ubuntu / Microk8s / K8S 1.21.7

Image of error: enter image description here

Answer 1

Check your PersistentVolumeClaims (kubectl get pvc).

The "has unbound immediate PersistentVolumeClaim" message suggests that your PVC status is "Pending". Meaning that either you don't have StorageClass "do-block-storage", or that the provisioner corresponding to that class did not create the underlying volume and corresponding PersistentVolume object.

Check your StorageClasses (kubectl get sc)

Make sure the storageClassName in your StatefulSets volumeClaimTemplate refers to an existing StorageClass.

Make sure the provisioner for that StorageClass works as expected (kubectl logs).

Alternatively, for a test, you could use ephemeral storage instead - remove the volumeClaimTemplate, add some emptyDir volume instead.

Answer 2

Your StatefulSet is referring to a non-existence StorageClass.

 ...
 volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch-logging
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: do-block-storage <--- change this to "data"
      resources:
        requests:
          storage: 50Gi