Cancel or undo deletion of Persistent Volumes in kubernetes cluster

Question

Accidentally tried to delete all PV\'s in cluster but thankfully they still have PVC\'s that are bound to them so all PV\'s are stuck in Status: Terminating.

How can I g

Answer 1

Edit: This only applies if you deleted the PVC and not the PV. Do not follow these instructions if you deleted the PV itself or the disk may be deleted!

I found myself in this same situation due to a careless mistake. It was with a statefulset on Google Cloud/GKE. My PVC said terminating because the pod referencing it was still running and the PV was configured with a retain policy of Deleted. I ended up finding a simpler method to get everything straightened out that also preserved all of the extra Google/Kubernetes metadata and names.

First, I would make a snapshot of your disk as suggested by another answer. You won't need it, but if something goes wrong, the other answer here can then be used to re-create a disk from it.

The short version is that you just need reconfigure the PV to "Retain", allow the PVC to get deleted, then remove the previous claim from the PV. A new PVC can then be bound to it and all is well.

Details:

1. Find the full name of the PV:

    kubectl get pv

2. Reconfigure your PV to set the reclaim policy to "Retain": (I'm doing this on Windows so you may need to handle the quotes differently depending on OS)

    kubectl patch pv  -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"

3. Verify that the status of the PV is now Retain.
4. Shutdown your pod/statefulset (and don't allow it to restart). Once that's finished, your PVC will get removed and the PV (and the disk it references!) will be left intact.
5. Edit the PV:

    kubectl edit pv 

6. In the editor, remove the entire "claimRef" section. Remove all of the lines from (and including) "claimRef:" until the next tag with the same indentation level. The lines to remove should look more or less like this:

      claimRef:
        apiVersion: v1
        kind: PersistentVolumeClaim
        name: my-app-pvc-my-app-0
        namespace: default
        resourceVersion: "1234567"
        uid: 12345678-1234-1234-1234-1234567890ab

7. Save the changes and close the editor. Check the status of the PV and it should now show "Available".
8. Now you can re-create your PVC exactly as you originally did. That should then find the now "Available" PV and bind itself to it. In my case, I have the PVC defined with my statefulset as a volumeClaimTemplate so all I had to do was "kubectl apply" my statefulset.