Pod not terminating

Question

We have a Rancher Kubernetes cluster where sometimes the pods get stuck in terminating status when we try to delete the corresponding deployment, as shown below.

$ kubectl get deployments NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE ... storage-manager-deployment 1 0 0 0 1d ... $ kubectl delete deployments storage-manager-deployment kubectl delete deployments storage-manager-deployment deployment.extensions "storage-manager-deployment" deleted Cc Cc^C $ kubectl get po NAME READY STATUS RESTARTS AGE ... storage-manager-deployment-6d56967cdd-7bgv5 0/1 Terminating 0 23h ... $ kubectl delete pods storage-manager-deployment-6d56967cdd-7bgv5 --grace-period=0 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "storage-manager-deployment-6d56967cdd-7bgv5" force deleted Cc Cc^C

Both the delete commands (for the deployment and the pod ) get stuck and need to be stopped manually.

We have tried both

kubectl delete pod NAME --grace-period=0 --force

and

kubectl delete pod NAME --now

without any luck.

We have also set fs.may_detach_mounts=1 , so it seems that all the similar questions already on StackOverflow don't apply to our problem.

If we check the node on which the incriminated pod runs, it does not appear in the docker ps list.

Any suggestion?

Thanks

Answer 1

Check the pod spec for an array: 'finalizers'

finalizers:
- cattle-system

If this exists, remove it, and the pod will terminate.