Kubernetes系列之监控Metres-server实战篇

本次系列使用的所需部署包版本都使用的目前最新的或最新稳定版，安装包地址请到公众号内回复【K8s实战】获取

介绍

从 Kubernetes 1.8 开始，Kubernetes 通过 Metrics API 获取资源使用指标，例如容器 CPU 和内存使用情况。这些度量指标可以由用户直接访问，例如通过使用kubectl top 命令，或者使用集群中的控制器。

Metrics API: 通过 Metrics API，您可以获得 node 或 pod 当前的资源使用情况（但是不存储）。

大致是说它符合 kubernetes 的监控架构设计，受 heapster 项目启发，并且比 heapster 优势在于：

访问不需要 apiserver 的代理机制，提供认证和授权等;

很多集群内组件依赖它（HPA,scheduler,kubectl top），因此它应该在集群中默认运行;

下载编排

[root@master-01 opt]# git clone https://github.com/kubernetes-incubator/metrics-server[root@master-01 opt]# cd metrics-server/deploy/1.8+/

创建metrics-server证书

创建签名请求

[root@master-01 1.8+]# cd /etc/kubernetes/ssl/[root@master-01 ssl]# cat > metrics-server-csr.json <<EOF{"CN": "aggregator","hosts": [],"key": {"algo": "rsa","size": 2048},"names": [{"C": "CN","ST": "Hangzhou","L": "Hangzhou","O": "k8s","OU": "4Paradigm"}]}EOF

创建证书和私钥

[root@master-01 ssl]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes metrics-server-csr.json|cfssljson -bare metrics-server2019/03/13 15:23:01 [INFO] generate received request2019/03/13 15:23:01 [INFO] received CSR2019/03/13 15:23:01 [INFO] generating key: rsa-20482019/03/13 15:23:01 [INFO] encoded CSR2019/03/13 15:23:01 [INFO] signed certificate with serial number 1026675139058810263099374133507485748972230132012019/03/13 15:23:01 [WARNING] This certificate lacks a "hosts" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 ("Information Requirements").

同步证书

同步证书到master-2、master-03

[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.131:/etc/kubernetes/ssl/[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.132:/etc/kubernetes/ssl/

开启聚合配置

修改kube-apiserver配置文件来支持metres-server，加入如下启动参数来启用aggregation layer：

--proxy-client-cert-file=/etc/kubernetes/ssl/metrics-server.pem \--proxy-client-key-file=/etc/kubernetes/ssl/metrics-server-key.pem \--runtime-config=api/all=true \--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \--requestheader-allowed-names=aggregator \--requestheader-extra-headers-prefix=X-Remote-Extra- \--requestheader-group-headers=X-Remote-Group \--requestheader-username-headers=X-Remote-User

--requestheader-XXX、--proxy-client-XXX 是 kube-apiserver 的 aggregator layer 相关的配置参数，metrics-server & HPA 需要使用。

--requestheader-client-ca-file：用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书；在启用了 metric aggregator 时使用。

如果 --requestheader-allowed-names 不为空，则--proxy-client-cert-file 证书的 CN 必须位于 allowed-names 中，默认为 aggregator。

注意：需要重启三台主控的kube-apiserver

如果不开启聚合配置可能会报如下错误

这是因为没用开启聚合层

I0313 05:18:36.447202 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)Error: cluster doesn't provide requestheader-client-ca-file

修改编排文件

在metrics-server-deployment.yaml文件中containers字段下添加如下

command:- /metrics-server- --kubelet-insecure-tls- --kubelet-preferred-address-types=InternalIP

如果不添加稍后可能会报如下错误

E0313 08:23:41.193222 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.209.130: unable to fetch metrics from Kubelet 192.168.209.130 (192.168.209.130): Get https://192.168.209.130:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.131: unable to fetch metrics from Kubelet 192.168.209.131 (192.168.209.131): Get https://192.168.209.131:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.132: unable to fetch metrics from Kubelet 192.168.209.132 (192.168.209.132): Get https://192.168.209.132:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.133: unable to fetch metrics from Kubelet 192.168.209.133 (192.168.209.133): Get https://192.168.209.133:10250/stats/summary/: x509: certificate signed by unknown authority]

资料

https://github.com/kubernetes-incubator/metrics-server/issues/67

https://github.com/mattkelly/metrics-server/commit/bfddc174c783290cb86d6da2fe1182d53a3b9bd5

gcr.io的镜像访问不到的话需要将metrics-server-deployment.yaml中的镜像替换为：registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1

创建metrics-server

[root@master-01 1.8+]# kubectl apply -f ./

查看服务状态

[root@master-01 1.8+]# kubectl get pod -nkube-systemNAME READY STATUS RESTARTS AGEmetrics-server-7c499cd69d-499js 1/1 Running 0 14s

测试功能

可以看到资源使用信息采集到了

[root@master-01 1.8+]# kubectl top pods --all-namespacesNAMESPACE NAME CPU(cores) MEMORY(bytes) default dnstools-6b77cc4988-b5smz 0m 2Mi default nginx-7899755b7-rgdch 0m 2Mi default tests-mychart-7d84ff968f-76d2l 1m 3Mi default wordpress-test-mariadb-59cfd7c475-27chl 5m 116Mi default wordpress-test-wordpress-6fc9b7cc7f-b2nfq 4m 149Mi ingress-nginx grafana-69549786b6-d78nv 1m 30Mi ingress-nginx prometheus-server-8658d8cdbb-4qps2 1m 20Mi kube-system coredns-5d668bd598-4xxwn 3m 13Mi kube-system coredns-5d668bd598-f5g96 2m 9Mi kube-system kubernetes-dashboard-cb55bd5bd-gc84g 1m 19Mi kube-system metrics-server-84f9775b88-gh7x7 2m 16Mi kube-system tiller-deploy-87d7c6dfb-kxj7p 1m 9Mi monitoring kube-state-metrics-6f8967c6c5-nzkxp 2m 30Mi monitoring node-exporter-4n9wj 1m 8Mi monitoring node-exporter-5wtgw 0m 8Mi monitoring node-exporter-gdj8f 1m 11Mi monitoring node-exporter-p96zj 1m 9Mi monitoring prometheus-operator-795895d784-v569s 1m 10Mi [root@master-01 1.8+]# kubectl top nodesNAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.209.130 158m 7% 1965Mi 53% 192.168.209.131 103m 5% 1859Mi 50% 192.168.209.132 123m 6% 2152Mi 58% 192.168.209.133 38m 1% 1022Mi 27%

通过 kube-apiserver接口访问

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes/

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/namespace/pods/

[root@master-01 1.8+]# curl -k https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods {"kind": "PodMetricsList","apiVersion": "metrics.k8s.io/v1beta1","metadata": {"selfLink": "/apis/metrics.k8s.io/v1beta1/pods"},"items": [{"metadata": {"name": "coredns-5d668bd598-f5g96","namespace": "kube-system","selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5d668bd598-f5g96","creationTimestamp": "2019-03-13T09:23:58Z"},"timestamp": "2019-03-13T09:23:17Z","window": "30s","containers": [{"name": "coredns","usage": {"cpu": "2073503n","memory": "9628Ki"}...............

浏览器访问

这里的30000端口是我把metrics-server端口改成NodePort了。