[root@szy-k8s-master log]# kubectl describe pod prometheus-544bf54848-b97rn --namespace=kube-system Name: prometheus-544bf54848-b97rn Namespace: kube-system Node: szy-k8s-node1/10.10.31.203 Start Time: Tue, 26 Jun 2018 17:05:08 +0800 Labels: app=prometheus pod-template-hash=1006910404 Annotations: <none> Status: Pending IP: Controlled By: ReplicaSet/prometheus-544bf54848 Containers: prometheus: Container ID: Image: prom/prometheus:latest Image ID: Port: 9090/TCP Host Port: 0/TCP State: Waiting Reason: ContainerCreating Ready: False Restart Count: 0 Environment: <none> Mounts: /etc/prometheus/prometheus.yml from prometheus-conf-volume (rw) /etc/prometheus/rules from prometheus-rules-volume (rw) /prometheus from prometheus-data-volume (rw) /var/run/secrets/kubernetes.io/serviceaccount from prometheus-token-6lmsv (ro) Conditions: Type Status Initialized True Ready False PodScheduled True Volumes: prometheus-data-volume: Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace) ClaimName: prometheus-data-pvc ReadOnly: false prometheus-conf-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: prometheus-conf Optional: false prometheus-rules-volume: Type: ConfigMap (a volume populated by a ConfigMap) Name: prometheus-rules Optional: false prometheus-token-6lmsv: Type: Secret (a volume populated by a Secret) SecretName: prometheus-token-6lmsv Optional: false QoS Class: BestEffort Node-Selectors: <none> Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 30m (x348 over 16h) kubelet, szy-k8s-node1 Unable to mount volumes for pod "prometheus-544bf54848-b97rn_kube-system(068ba024-7920-11e8-89d1-00505681317e)": timeout expired waiting for volumes to attach or mount for pod "kube-system"/"prometheus-544bf54848-b97rn". list of unmounted volumes=[prometheus-data-volume]. list of unattached volumes=[prometheus-data-volume prometheus-conf-volume prometheus-rules-volume prometheus-token-6lmsv] Warning FailedMount 36s (x573 over 16h) kubelet, szy-k8s-node1 (combined from similar events): MountVolume.SetUp failed for volume "prometheus-data-pv" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/068ba024-7920-11e8-89d1-00505681317e/volumes/kubernetes.io~nfs/prometheus-data-pv --scope -- mount -t nfs 10.10.31.202:/nfs/prometheus/data /var/lib/kubelet/pods/068ba024-7920-11e8-89d1-00505681317e/volumes/kubernetes.io~nfs/prometheus-data-pv Output: Running scope as unit run-26056.scope. mount.nfs: requested NFS version or transport protocol is not supported解决方法：搭建NFS服务
#在master安装 [root@szy-k8s-master /]# yum install -y nfs-utils rpcbind #在每个node安装 [root@szy-k8s-node1 /]# yum install -y nfs-utils #在每个node安装 [root@szy-k8s-node2 /]# yum install -y nfs-utils 配置：
#NFS服务的配置文件 /etc/exports。 这个文件可能不会存在，需要新建 [root@szy-k8s-master /]# cat /etc/exports /nfs/prometheus/data/ 10.10.31.0/24(rw,no_root_squash,no_all_squash,sync) #exports中的配置的内容，需要创建下/nfs/prometheus/data/ [root@szy-k8s-master /]# mkdir -p /nfs/prometheus/data/ #修改权限 [root@szy-k8s-master /]# chmod -R 777 /nfs/prometheus/data/ #验证配置的/nfs/prometheus/data/是否正确 [root@szy-k8s-master /]# exportfs -r10.10.31.0/24：这个是运行访问NFS的IP范围，也就是10.10.31开头的IP，24是掩码长度。 根据自己的k8s主机网段设置。 (rw,no_root_squash,no_all_squash,sync)： 可以设定的参数主要有以下这些： rw：可读写的权限； ro：只读的权限； no_root_squash：登入到NFS主机的用户如果是root，该用户即拥有root权限； root_squash：登入NFS主机的用户如果是root，该用户权限将被限定为匿名使用者nobody； all_squash：不管登陆NFS主机的用户是何权限都会被重新设定为匿名使用者nobody。 anonuid：将登入NFS主机的用户都设定成指定的user id，此ID必须存在于/etc/passwd中。 anongid：同anonuid，但是变成group ID就是了！ sync：资料同步写入存储器中。 async：资料会先暂时存放在内存中，不会直接写入硬盘。 insecure：允许从这台机器过来的非授权访问。启动服务：
[root@szy-k8s-master /]# systemctl start rpcbind [root@szy-k8s-master /]# systemctl status rpcbind ● rpcbind.service - RPC bind service Loaded: loaded (/usr/lib/systemd/system/rpcbind.service; disabled; vendor preset: enabled) Active: active (running) since Wed 2018-06-27 11:05:29 CST; 1h 51min ago Process: 4111 ExecStart=/sbin/rpcbind -w $RPCBIND_ARGS (code=exited, status=0/SUCCESS) Main PID: 4112 (rpcbind) Memory: 580.0K CGroup: /system.slice/rpcbind.service └―4112 /sbin/rpcbind -w Jun 27 11:05:29 szy-k8s-master systemd[1]: Starting RPC bind service... Jun 27 11:05:29 szy-k8s-master systemd[1]: Started RPC bind service. [root@szy-k8s-master /]# systemctl start nfs [root@szy-k8s-master /]# systemctl status nfs ● nfs-server.service - NFS server and services Loaded: loaded (/usr/lib/systemd/system/nfs-server.service; disabled; vendor preset: disabled) Active: active (exited) since Wed 2018-06-27 11:26:54 CST; 1h 30min ago Process: 11859 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS) Process: 11853 ExecStartPre=/bin/sh -c /bin/kill -HUP `cat /run/gssproxy.pid` (code=exited, status=0/SUCCESS) Process: 11851 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS) Main PID: 11859 (code=exited, status=0/SUCCESS) Memory: 0B CGroup: /system.slice/nfs-server.service Jun 27 11:26:54 szy-k8s-master systemd[1]: Starting NFS server and services... Jun 27 11:26:54 szy-k8s-master systemd[1]: Started NFS server and services.检验：
[root@szy-k8s-master /]# showmount -e 10.10.31.202 Export list for 10.10.31.202: /nfs/prometheus/data 10.10.31.0/24 [root@szy-k8s-master /]# NFS客户端的操作： 1、showmout命令对于NFS的操作和查错有很大的帮助，所以我们先来看一下showmount的用法 showmout -a ：这个参数是一般在NFS SERVER上使用，是用来显示已经mount上本机nfs目录的cline机器。 -e ：显示指定的NFS SERVER上export出来的目录。 2、mount nfs目录的方法： mount -t nfs hostname(orIP):/directory /mount/point 再次查看问题，已经是成功，可挂载了。
[root@szy-k8s-master /]# kubectl describe pod prometheus-544bf54848-b97rn --namespace kube-system QoS Class: BestEffort Node-Selectors: <none> Tolerations: node-role.kubernetes.io/master:NoSchedule node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedMount 36m (x664 over 19h) kubelet, szy-k8s-node1 (combined from similar events): MountVolume.SetUp failed for volume "prometheus-data-pv" : mount failed: exit status 32 Mounting command: systemd-run Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/068ba024-7920-11e8-89d1-00505681317e/volumes/kubernetes.io~nfs/prometheus-data-pv --scope -- mount -t nfs 10.10.31.202:/nfs/prometheus/data /var/lib/kubelet/pods/068ba024-7920-11e8-89d1-00505681317e/volumes/kubernetes.io~nfs/prometheus-data-pv Output: Running scope as unit run-17572.scope. mount.nfs: access denied by server while mounting 10.10.31.202:/nfs/prometheus/data Warning FailedMount 20m (x412 over 19h) kubelet, szy-k8s-node1 Unable to mount volumes for pod "prometheus-544bf54848-b97rn_kube-system(068ba024-7920-11e8-89d1-00505681317e)": timeout expired waiting for volumes to attach or mount for pod "kube-system"/"prometheus-544bf54848-b97rn". list of unmounted volumes=[prometheus-data-volume]. list of unattached volumes=[prometheus-data-volume prometheus-conf-volume prometheus-rules-volume prometheus-token-6lmsv] Normal SuccessfulMountVolume 15m kubelet, szy-k8s-node1 MountVolume.SetUp succeeded for volume "prometheus-data-pv" Normal Pulled 10m kubelet, szy-k8s-node1 Successfully pulled image "prom/prometheus:latest" [root@szy-k8s-master /]#