Adding a volume to a Kubernetes StatefulSet using kubectl patch

Question

Problem summary:

I am following the Kubernetes guide to set up a sample Cassandra cluster . The cluster is up and running, and I would like to add a second volume to each node in order to try enable backups for Cassandra that would be stored on a separate volume.

My attempt to a solution:

I tried editing my cassandra-statefulset.yaml file by adding a new volumeMounts and volumeClaimTemplates entry, and reapplying it, but got the following error message:

$ kubectl apply -f cassandra-statefulset.yaml 
storageclass.storage.k8s.io/fast unchanged
The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy' are forbidden

I then tried to enable rolling updates and patch my configuration following the documentation here: https://kubernetes.io/docs/tasks/run-application/update-api-object-kubectl-patch/

$ kubectl patch statefulset cassandra -p '{"spec":{"updateStrategy":{"type":"RollingUpdate"}}}'
statefulset.apps/cassandra patched (no change)

My cassandra-backup-patch.yaml :

spec:
  template:
    spec:
      containers:
        volumeMounts:
        - name: cassandra-backup
          mountPath: /cassandra_backup
  volumeClaimTemplates:
  - metadata:
      name: cassandra-backup
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: fast
      resources:
        requests:
          storage: 1Gi

However this resulted in the following error:

$ kubectl patch statefulset cassandra --patch "$(cat cassandra-backup-patch.yaml)"
The request is invalid: patch: Invalid value: "map[spec:map[template:map[spec:map[containers:map[volumeMounts:[map[mountPath:/cassandra_backup name:cassandra-backup]]]]] volumeClaimTemplates:[map[metadata:map[name:cassandra-backup] spec:map[accessModes:[ReadWriteOnce] resources:map[requests:map[storage:1Gi]] storageClassName:fast]]]]]": cannot restore slice from map

Could anyone please point me to the correct way of adding an additional volume for each node or explain why the patch does not work? This is my first time using Kubernetes so my approach may be completely wrong. Any comment or help is very welcome, thanks in advance.

Answer 1

The answer is in your first log:

The StatefulSet "cassandra" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', and 'updateStrategy'

You can't change some fields in a statefulset after creation. You will likely need to delete and recreate the statefulset to add a new volumeClaimTemplate .

edit: It can many times be useful to leave your pods running even when you delete the statefulset . To accomplish this use the --cascade=false flag on the delete operation.

kubectl delete statefulset <name> --cascade=false

Then your workload will stay running while you recreate your statefulset with the updated VPC.

Answer 2

As mentioned by switchboard.op , deleting is the answer.

but

Watch out for deleting these objects:

• PersistentVolumeClaim ( kubectl get pvc )
• PersistentVolume ( kubectl get pv )

which for example in case you'd want to do just helm uninstall instead of kubectl delete statefulset/<item> will be deleted thus unless there's any other reference for the volumes and in case you don't have backups of the previous YAMLs that contain the IDs (ie not just generated from Helm templates, but from the orchestrator) you might have a painful day ahead of you.

PVCs and PVs hold IDs and other reference properties for the underlying (probably/mostly?) vendor specific volume referencing by eg S3 or other object or file storage implementation used in the background as a volume in a Pod or other resources.

Deleting or otherwise modifying a StatefulSet if you preserve the PVC name within the spec doesn't affect mounting of the correct resource.

If in doubt, always just copy locally the whole volume prior to doing destructive action to PVCs and PVs if you need them in the future or running commands without knowing the underlying source code eg by:

kubectl cp <some-namespace>/<some-pod>:/var/lib/something /tmp/backup-something

and then just load it back by reversing the arguments.

Also for Helm usage, delete the StatefulSet , then issue helm upgrade command and it'll fix the missing StatefulSet without touching PVCs and PVs.