How to manual recover a PV

Question

according to the official docs https://kubernetes.io/docs/tasks/administer-cluster/change-pv-reclaim-policy/ with the “Retain” policy a PV can be manually recovered . What d

Answer 1

The process to manually recover the volume is as below.

You can use the same PV to mount to different pod along with the data even after the PVC is deleted (PV must exist, will typically exist if the reclaim policy of storageclass is Retain)

Verify that PV is in released state. (ie no pvc has claimed it currently)

 ➜  ~ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                     STORAGECLASS   REASON   AGE
pvc-eae6acda-59c7-11e9-ab12-06151ee9837e   16Gi       RWO            Retain           Released   default/dhanvi-test-pvc   gp2                     52m

Edit the PV (kubectl edit pv pvc-eae6acda-59c7-11e9-ab12-06151ee9837e) and remove the spec.claimRef part. The PV claim would be unset like below.

 ➜  ~ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   REASON   AGE
pvc-eae6acda-59c7-11e9-ab12-06151ee9837e   16Gi       RWO            Retain           Available           gp2                     57m

Then claim the PV using PVC as below.

---

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: dhanvi-test-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 16Gi
  volumeName: "pvc-eae6acda-59c7-11e9-ab12-06151ee9837e"

Can be used in the pods as below.

volumes:
- name: dhanvi-test-volume
  persistentVolumeClaim:
    claimName: dhanvi-test-pvc

Update: Volume cloning might help https://kubernetes.io/blog/2019/06/21/introducing-volume-cloning-alpha-for-kubernetes/

Answer 2

There are three reclaim policies which define what happens with the persistent volume after deletion of the bound volume claim

• Retain
• Delete
• Recycle

Delete means the persistent volume as well as the associated storage asset in the external infrastructure is deleted.

Recycle will clean up the volume rm -rf /thevolume/* and after that it will be available for new persistent volume claims.

Retain leaves persistent volume in state released which does not allow for new persistent volume claims to reclaim it. The whole reclaim process is manual. You need to delete the persistent volume yourself. You can backup the data from the storage asset and delete the data afterwards. Then you can either delete the storage asset or create a new persistent volume for this asset.

If you want to write the data to another persistent volume using Kubernetes you could use a Job to copy the data.

In that case make sure you use persistent volume access modes ROX - ReadOnlyMany or RWX - ReadWriteMany and start a Job running a container which claims the persistent volume to be backed-up using a selector and claim another destination backup volume. Then copy the data via the container.

Alternatively, you can do the backup outside Kubernetes. Your method does then depend on the type of storage asset you are using. E.g., if you are using NFS you could mount source and destination and copy the data via command line.

Both options I've framed are more or less manual backup strategy. If you aim for a more sophisticated backup strategy for production workloads you might have a look at Stash - Backup for your disks for production workloads in Kubernetes

Answer 3

Like stated in the answer by Tummala Dhanvi, the spec.claimRef section has to be tackled with. While removing the whole spec.claimRef can be useful if you have only one PV, this will prove very nasty if you have multiple PVs to be "rescued".

The first step is to ensure the PV has the Retain reclaim policy before deleting the PVC. You can edit or patch the PV to achieve that

• kubectl edit pv pvc-73e9252b-67ed-4350-bed0-7f27c92ce826
  • find the spec.persistentVolumeReclaimPolicy key
  • input Retain for its value
  • save & exit
• or, in one command kubectl patch pv pvc-73e9252b-67ed-4350-bed0-7f27c92ce826 -p "{\"spec\":{\"persistentVolumeReclaimPolicy\":\"Retain\"}}"

Now you can delete the PVC(s) (either by using helm or otherwise) and the PV(s) will not be deleted.

To successfully re-mount a PV to the desired pod you have to edit the PV configuration once again, this time the spec.claimRef section. But do not delete the whole section. Instead, delete only the resourceVersion and uid keys. The resulting section would then look something like this

...
  capacity:
    storage: 16Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: database
    namespace: staging
  nodeAffinity:
...

Repeat this for all of your PVs and their status in the kubectl get pv output will be Available afterwards. By leaving the spec.claimRef.name and spec.claimRef.namespace keys intact, we ensured that a new PVC with the corresponding spec (staging/database in my case), will be bound to the exact PV it is supposed to.

Also, make sure your new claim does not specify a larger storage capacity than the PV actually has (it seems though that the new claims' capacity may be less than the existing PV's). If the new PVC claims a larger storage, a new PV will be created instead. Best to keep it the same.

To digress: If the storageClass you're using allows for volume resizing, you can resize it later; here it's explained how: https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/

My experience with this was pretty stressful. I've had 6 PVs, thakfully in Retain mode. For some reason a new deployment rollout got stuck, two pods just wouldn't want to terminate. In the end I ended up with deleting the whole deployment (using helm), restarting the cluster nodes, and then redeploying anew. This caused 6 new PVs to be created!

I found this thread, and went on to delete the spec.claimRef of all the PVs. Deleting and deploying the installation once again resulted in the PVs being reused, but they were not mounted where they supposed to, the data was not there. After a good amount of digging, I figured out that the database volume was mounted to a RabbitMQ pod, mongodb was mounted to ElasticSearch etc.

It took me about a dozen times around, to get this right. Luckily, for me the mixed-up mounting of volumes did not destroy any of the original data. The pods initializations did not clean-out the volume, just wrote their stuff there.

Hope this saves some seriuos headaches out there!