Change Kubernetes Persistent Volume's Storage Class and keep the data

Question

I have Elasticsearch Data pods that are currently running on an AKS and are connected to Persistent Volumes that is using a Premium SSD Managed Disk Storage Class and I want to downgrade it to Standard SSD Managed Disk without losing the data I have on the currently used Persistent Volume. I've created a new Storage Class that is defined with Standard SSD Managed Disk, but if I create a new PV from that it obviously doesn't keep the old data and I need to copy it somehow, so I was wondering what would be best practice switching PV's Storage Class.

Answer 1

Unfortunately, once a PVC is created and a PV is provisioned for it, the only thing you can change without creating a new one is the volume's size

The only straightforward way I could think of without leveraging CSI snapshots/clones, which you might not have access to (depends on how you created PVCs/PVs AFAIK), would be to create a new PVC and mount both volumes on a Deployment whose Pod has root access and the rsync command.

Running rsync -a /old/volume/mount/path /new/volume/mount/path on such a Pod should get you what you want.

However, you should make sure that you do so BEFORE deleting PVCs or any other resource using your PVs. By default, most of the default storage classes create volumes with reclaim policies that immediately delete the PV as soon as all resources using it are gone, so there's a small risk of data loss

Answer 2

It is not possible in Kube.netes to change the storage class of a PVC - because of the described reason. Unfortunately you have to create a new PVC with the desired storage class.

Answer 3

One way to go about this would be to create a snapshot and then new volume from the that snapshot: https://kube.netes.io/docs/concepts/storage/volume-snapshots/

Another is to leverage CSI cloning: https://kube.netes.io/docs/concepts/storage/volume-pvc-datasource/

Both effectively facilitate creation of a fresh PV with a copy of existing data

Answer 4

In extension to @LeoD's answer, I'm sharing a complete step by step guide of what I did exactly eventually.

1. Scale Stateful Set to 0.

2. Find the old PVs' Disk resources in Azure:

• kubectl describe pv to check which PVs are associated with the PVCs I want to replace.
• Follow the resource ID that is specified in field DiskURI to get to the Disks in Azure.

3. Create an Azure Snapshot resource from each PVC Disk.

4. Create a new Disk with at least the same allocated GiB as current PVC has from the Snapshots. SKU still doesn't matter at this point but I preferred using the new SKU I want to implement.

5. Create a K8S Persistent Volume from each new Disk:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-from-azure-disk-0
  namespace: my-namespace
  annotations:
    pv.kubernetes.io/provisioned-by: disk.csi.azure.com
spec:
  capacity:
    storage: 1064
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  storageClassName: managed
  claimRef:
    name: pvc-from-azure-0
    namespace: my-namespace
  csi:
    driver: disk.csi.azure.com
    volumeHandle: /subscriptions/<Your subscription ID>/resourcegroups/<Name of RG where the new Disk is stored at>/providers/microsoft.compute/disks/new-disk-from-pvc-0
    fsType: ext4

Make sure to edit the following fields:

• metadata.namespace : To the namespace where you have your old resources.
• spec.capacity.storage : To the allocated GiB you chose for the new Disk.
• spec.storageClassName : Name of Storage Class that suits the SKU you chose for the Disk.
• spec.claimRef.name : Here we specify a name for a PVC we haven't created yet that will be used later, just make sure the name you choose here is the name you're going to use when creating the PVC in the next step.
• spec.claimRef.namespace : To the namespace where you have your old resources.
• csi.volumeHandle : Insert Resource ID of your new Disk.

6. Create a K8S Persistent Volume Claim from each new Persistent Volume you've just created:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-from-azure-0
  namespace: my-namespace
spec:
  storageClassName: managed
  volumeName: pv-from-azure-disk-0
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1064

Make sure to edit the following fields:

• metadata.name : Use the exact same name you used in the PV's spec.claimRef.name field.
• metadata.namespace : To the namespace where you have your old resources.
• spec.storageClassName : To the name of Storage Class that you specified for the PV in the spec.storageClassName field.
• spec.volumeName : To the name of the PV you've created in the former step.
• spec.resources.requests.storage : To the allocated GiB you specified in the PV's spec.capacity.storage field.

7. Perform steps 2-6 for each existing PVC you want to replace; if your Stateful Set had 3 Pods for example you should do that 3 times for each Pod's PVC.

8. Create an Ubuntu Pod for each new PVC and mount the PVC to the pod. (If you have 3 PVCs for example you should be having 3 Ubuntu pods, one for each).

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-0
  namespace: my-namespace
spec:
  containers:
   - name: ubuntu-0
     image: ubuntu
     command:
      - "sleep"
      - "604800"
     volumeMounts:
       - mountPath: /mnt/snapshot-data-0
         name: snapshot-data-0
  volumes:
   - name: snapshot-data-0
     persistentVolumeClaim:
       claimName: pvc-from-azure-0

Make sure to edit the following fields:

• metadata.namespace : To the namespace where you have your old resources.
• spec.volumes.persistentVolumeClaim.claimName : To the name of the new PVC you've created.

9. Exec to each Ubuntu Pod and validate all your data is in there.

10. If all data is there you can delete the Ubuntu Pod.

11. Now is the very crucial part, you need to delete the old original PVCs and Stateful Set but before you do that, make sure you have the original yaml file of the Stateful Set or anything else you used to create the Stateful Set, make sure you know how to re-create the Stateful Set and only then continue to delete the old original PVCs and Stateful Set.

12. After deletion of Stateful Set and old PVCs has been completed, in your Stateful Set yaml or configuration replace the value of storageClassName that's within the volumeClaimTemplates block to the new Storage Class you want to use instead.

...

volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  storageClassName: "managed" # <--- Edit this field
  resources:
    requests:
      storage: "1064"
    limits:
      storage: "1064"
      
...      

13. Now recreate the Stateful Set with the new configuration. This will recreate the Stateful Set and create new empty PVCs which are going to use the new specified SKU from the Storage Class.

14. After creation of the PVCs has been completed, scale the new Stateful Set to 0.

15. Now recreate the Ubuntu Pod but this time mount both the PVC from the Azure Disk you created in the first steps and the new empty PVC of the new Stateful Set:

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-0
  namespace: my-namespace
spec:
  containers:
   - name: ubuntu-0
     image: ubuntu
     command:
      - "sleep"
      - "604800"
     volumeMounts:
       - mountPath: /mnt/snapshot-data-0
         name: snapshot-data-0
       - mountPath: /mnt/new-data-0
         name: new-data-0           
  volumes:
   - name: snapshot-data-0
     persistentVolumeClaim:
       claimName: pvc-from-azure-0
   - name: new-data-0
     persistentVolumeClaim:
       claimName: my-stateful-set-data-0

Make sure to edit the following fields:

• metadata.namespace : To the namespace where you have your resources.
• spec.volumes.persistentVolumeClaim : There are two fields of this, one for snapshot-data-0 , and one for new-data-0 . As their names suggest, the snapshot-data-0 should have the old data from the Azure Disk we created in the first steps, and the new-data-0 should be empty since it should be the new PVC that was created from the recreation of the Stateful Set. So, the value for spec.volumes.persistentVolumeClaim of snapshot-data-0 should be the name of the new PVC you've created from the Azure Disk. And the value for spec.volumes.persistentVolumeClaim of new-data-0 should be the name of the new PVC that was created following the recreation of the Stateful Set.

16. Exec to the Ubuntu Pod and perform the following commands:

apt update && apt install rsync -y
nohup rsync -avz /mnt/snapshot-data-0/ /mnt/new-data-0 --log-file=$HOME/.rsyncd.log &

First command is to install rsync and second command to copy all the old data into the new empty PVC and keep logs of the command at /root/.rsyncd.log for your reference of checking the progress. Do notice depending on the amount of data you have stored that this command could take some time. For me personally it took about 1 hour to complete.

17. Perform steps 15-16 for each backed up and old PVC pair.

18. Once it finishes for all PVC pairs, delete the Ubuntu Pods.

19. Scale the new Stateful Set back up.

20. Profit. You now have the same Stateful Set having the same data but with a different Storage Class.