Kubernetes on GKE can't mount volumes

Question

There are two nodes and 2 pods running in my cluster (1 pod on each node) My persistent volume claim is below

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: blockchain-data
  annotations: {
        "volume.beta.kubernetes.io/storage-class": "blockchain-disk"
    }
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: ssd
  resources:
    requests:
      storage: 500Gi

and mystorageclass

kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: blockchain-disk
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd

and I mounted it on my container like this

spec:
      containers:
        - image: gcr.io/indiesquare-dev/geth-node:v1.8.12
          imagePullPolicy: IfNotPresent
          name: geth-node
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - name: blockchain-data
              mountPath: /root/.ethereum
      volumes:
        - name: blockchain-data
          persistentVolumeClaim: 
            claimName: blockchain-data  

I have replicas set to 2. When start the deployment, the first pod starts correctly with the disk properly mounted.

However, the second pod gets stuck at containtercreating

If I run kubectl describe pods

Warning  FailedAttachVolume     18m               attachdetach-controller                       Multi-Attach error for volume "pvc-c56fbb79-954f-11e8-870b-4201ac100003" Volume is already exclusively attached to one node and can't be attached to another

I think according to this message, I am trying to attach the disk which is already attached to another node.

What I want to do is to have two persistent volumes separately attached to two pods. If the pods scale up, then each should have a different volume attached. How can I do this?

Answer 1

You can't attach a GCE Persistent Disk to multiple nodes. So if your pods are landing on different nodes you can't reuse the same disk.

You need something like ReadOnlyMany access mode but you have ReadWriteOnce.

Read https://cloud.google.com/kubernetes-engine/docs/concepts/persistent-volumes#access_modes