无法正确删除cStor卷副本

Question

I did ctrl-c terminate the kubectl delete -f openebs-operator.yml because it hangs definitely while the openebs is empty already. didn't realize the background jobs are still running.

get this message in kube-apiserver log:
E0112 18:10:53.877564 999 crd_finalizer.go:275] cstorvolumereplicas.openebs.io failed with: timed out waiting for the condition

Looks like cstorvolumereplicas.openebs.io cannot be deleted successfully. It says delete successfully but still there

Answer 1

The proper way of graceful deletion on OpenEBS is mentioned in OpenEBS docs under uninstallation section.

The recommended steps are:

Delete all the OpenEBS PVCs that were created. You can check the status of PVC using the following command

kubectl get pvc

There should not have any entries of OpenEBS PVC.

Delete all the SPCs (In case of cStor storage engine)

kubectl get spc -n openebs

There should not have any entries of OpenEBS SPC.

Ensure that no volume or pool pods are pending in terminating state . You can check the running status of Pods using the following command.

kubectl get pods -n <openebs namespace>

Delete the OpenEBS namespace either via helm purge or kubectl delete ns openebs.

Uninstalling the OpenEBS doesn't automatically delete the CRDs that were created. If you would like to complete remove the CRDs and the associated objects, run the following commands:

kubectl delete crd castemplates.openebs.io
kubectl delete crd cstorpools.openebs.io
kubectl delete crd cstorvolumereplicas.openebs.io
kubectl delete crd cstorvolumes.openebs.io
kubectl delete crd runtasks.openebs.io
kubectl delete crd storagepoolclaims.openebs.io
kubectl delete crd storagepools.openebs.io
kubectl delete crd volumesnapshotdatas.volumesnapshot.external-storage.k8s.io
kubectl delete crd volumesnapshots.volumesnapshot.external-storage.k8s.io

As part of deleting the Jiva Volumes, OpenEBS launches scrub jobs for clearing the data from the nodes. The completed jobs need to be cleared using the following command.

kubectl delete jobs -l openebs.io/cas-type=jiva -n <namespace>

Sometimes, there are chances that cStor volumes will not get deleted. Below workaround will resolve this issue, perform following :

kubectl edit cvr -n openebs

And then remove finalizers from the CVR Need to remove following entries

finalizers:
- cstorvolumereplica.openebs.io/finalizer