Kubernetes statefulset with NFS persistent volume

问题

I have a kubernetes cluster and I have a simple deployment for mongodb with NFS persistent volume set. It works fine, but since resources like databases are stateful I thought of using Statefulset for the mongodb, but now the problem is, when I go through the documentation, statefulset has volumeClaimTemplates instead of volumes (in deployments).

But now the problem comes.

in a deployment do it like this:

PersistentVolume -> PersistentVolumeClaim -> Deployment

But how can we do this in Statefulset ?

Is it like:

volumeClaimTemplates -> StatefulSet

How can I set a PersistentVolume for the volumeClaimTemplates. If we don't use PersistentVolume for StatefulSet, how does it create he volume and WHERE does it create the volumes? Is in host machines (i.e. kubernetes worker nodes)?

Because I have a separate NFS provisioner I am using for the mongodb deployment (with replicasset=1), how can I use the same setup with StatefulSet ?

Here's the my mongo-deployment.yaml -> which I am going to transform into a statefulset as shown in the second code snippet (mongo-stateful.yaml)

mongo-deployment.yaml

<omitted>
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs # name can be anything
spec:
  storageClassName: manual # same storage class as pvc
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: task-pv-claim
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany #  must be the same as PersistentVolume
  resources:
    requests:
      storage: 1Gi          
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: mongodb-deployment    
  labels:
    name: mongodb
spec:
  selector:
    matchLabels:
      app: mongodb
  replicas: 1
  template:
    metadata:
      labels: 
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo
        ports:
        -  containerPort: 27017
        ... # omitted some parts for easy reading
        volumeMounts:
        - name: data  
          mountPath: /data/db
      volumes: 
        - name: data
          persistentVolumeClaim: 
            claimName: task-pv-claim

mongo-stateful.yaml

---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: task-pv-volume
  labels:
    name: mynfs # name can be anything
spec:
  storageClassName: manual # same storage class as pvc
  capacity:
    storage: 10Gi
  accessModes:
    - ReadWriteOnce
  nfs:
    server: <nfs-server-ip>
    path: "/srv/nfs/mydata" 
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb-statefulset
spec:
  selector:
    matchLabels:
      name: mongodb-statefulset
  serviceName: mongodb-statefulset
  replicas: 2
  template:
    metadata:
      labels:
        name: mongodb-statefulset
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongodb
        image: mongo:3.6.4
        ports:
        - containerPort: 27017
        volumeMounts:
        - name: db-data
          mountPath: /data/db
  volumeClaimTemplates:
  - metadata:
      name: db-data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "manual"
      resources:
        requests:
          storage: 2Gi

But this is not working (mongo-stateful.yaml) pods are in pending state as when I describe it shows:

default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 pod has unbound immediate PersistentVolumeClaims

PS: Deployment works fine without any errors, problem is with Statefulset

Can someone please help me, how to write a statefulset with volumes?

回答1:

If your storage class does not support dynamic volume provisionning, you have to manually create PVs and associated PVCs, using yaml files, then the volumeClaimTemplates will allow to link existing PVCs with your statefulset's pods.

Here is a working example: https://github.com/k8s-school/k8s-school/blob/master/examples/MONGODB-install.sh

You should:

run it locally on https://kind.sigs.k8s.io/, which support dynamic volume provisionning, so here PVCs and PVs will be created automatically
export PVs and PVCs yaml files
use these yaml file as template to create your PVs and PVCs for your NFS backend.

Here is what you will get on Kind:

$ ./MONGODB-install.sh               
+ kubectl apply -f 13-12-mongo-configmap.yaml
configmap/mongo-init created
+ kubectl apply -f 13-11-mongo-service.yaml
service/mongo created
+ kubectl apply -f 13-14-mongo-pvc.yaml
statefulset.apps/mongo created
$ kubectl get pods
NAME      READY   STATUS    RESTARTS   AGE
mongo-0   2/2     Running   0          8m38s
mongo-1   2/2     Running   0          5m58s
mongo-2   2/2     Running   0          5m45s
$ kubectl get pvc
NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
database-mongo-0   Bound    pvc-05247511-096e-4af5-8944-17e0d8222512   1Gi        RWO            standard       8m42s
database-mongo-1   Bound    pvc-f53c35a4-6fc0-4b18-b5fc-d7646815c0dd   1Gi        RWO            standard       6m2s
database-mongo-2   Bound    pvc-2a711892-eeee-4481-94b7-6b46bf5b76a7   1Gi        RWO            standard       5m49s
$ kubectl get pv 
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                      STORAGECLASS   REASON   AGE
pvc-05247511-096e-4af5-8944-17e0d8222512   1Gi        RWO            Delete           Bound    default/database-mongo-0   standard                8m40s
pvc-2a711892-eeee-4481-94b7-6b46bf5b76a7   1Gi        RWO            Delete           Bound    default/database-mongo-2   standard                5m47s
pvc-f53c35a4-6fc0-4b18-b5fc-d7646815c0dd   1Gi        RWO            Delete           Bound    default/database-mongo-1   standard                6m1s

And a dump of a PVC (generated here by volumeClaimTemplate because odf kind dynamic volume provisionning):

$ kubectl get pvc database-mongo-0 -o yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rancher.io/local-path
    volume.kubernetes.io/selected-node: kind-worker2
  creationTimestamp: "2020-10-16T15:05:20Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app: mongo
  managedFields:
    ...
  name: database-mongo-0
  namespace: default
  resourceVersion: "2259"
  selfLink: /api/v1/namespaces/default/persistentvolumeclaims/database-mongo-0
  uid: 05247511-096e-4af5-8944-17e0d8222512
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: standard
  volumeMode: Filesystem
  volumeName: pvc-05247511-096e-4af5-8944-17e0d8222512
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

And the related PV:

kubectl get pv pvc-05247511-096e-4af5-8944-17e0d8222512 -o yaml     
apiVersion: v1
kind: PersistentVolume
metadata:
  annotations:
    pv.kubernetes.io/provisioned-by: rancher.io/local-path
  creationTimestamp: "2020-10-16T15:05:23Z"
  finalizers:
  - kubernetes.io/pv-protection
  managedFields:
    ...
  name: pvc-05247511-096e-4af5-8944-17e0d8222512
  resourceVersion: "2256"
  selfLink: /api/v1/persistentvolumes/pvc-05247511-096e-4af5-8944-17e0d8222512
  uid: 3d1e894e-0924-411a-8378-338e48ba4a28
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    apiVersion: v1
    kind: PersistentVolumeClaim
    name: database-mongo-0
    namespace: default
    resourceVersion: "2238"
    uid: 05247511-096e-4af5-8944-17e0d8222512
  hostPath:
    path: /var/local-path-provisioner/pvc-05247511-096e-4af5-8944-17e0d8222512_default_database-mongo-0
    type: DirectoryOrCreate
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - kind-worker2
  persistentVolumeReclaimPolicy: Delete
  storageClassName: standard
  volumeMode: Filesystem
status:
  phase: Bound

来源：https://stackoverflow.com/questions/64386094/kubernetes-statefulset-with-nfs-persistent-volume

标签

mongodb

Kubernetes

kubernetes-statefulset