服务器信息

类型	服务器IP地址	备注
Ansible(2台)	172.24.78.21/22	K8S集群部署服务器，可以和在一起，需要配置在负载均衡上实现反向代理，dashboard的端口为8443
K8S Master(2台)	172.24.78.21/22	K8s控制端，通过一个VIP做主备高可用
Harbor(2台)	172.24.78.23/24	高可用镜像服务器
Etcd(最少3台)	172.24.78.25/26/27	保存k8s集群数据的服务器
Hproxy(2台)	172.24.78.28/29	高可用etcd代理服务器
Node节点(2-N台)	172.24.78.31/32/xxx	真正运行容器的服务器，高可用环境至少两台

主机信息

序号	类型	服务器IP	主机名	VIP
1	K8S Master1	172.24.78.21	master1.his.net	172.24.78.18
2	K8S Master2	172.24.78.22	master2.his.net	172.24.78.18
3	Harbor1	172.24.78.23	harbor1.his.net
4	Harbor2	172.24.78.24	harbor2.his.net
5	etcd节点1	172.24.78.25	etcd1.his.net
6	etcd节点2	172.24.78.26	etcd2.his.net
7	etcd节点3	172.24.78.27	etcd3.his.net
8	Haproxy1	172.24.78.28	ha1.his.net
9	Haproxy2	172.24.78.29	ha2.his.net
10	Node节点1	172.24.78.12	node1.his.net
3	Node节点2	172.24.78.31	node2.his.net

软件信息

端口：192.168.7.248：6443 #需要配置在负载均衡上实现反向代理，dashboard的端口为8443
操作系统：ubuntu server 1804
k8s版本： 1.13.5
calico：3.4.4

修改主机名及IP地址

vim /etc/netplan/50-cloud-init.yaml
            dhcp4: no
            dhcp6: no
            addresses: [172.24.78.25/25]
            gateway4: 172.24.78.1
            nameservers:
                    addresses: [34.34.34.34,202.96.134.133]

hostnamectl set-hostname master1.his.net...

添加hosts映射

172.24.78.21  master1.his.net
172.24.78.22  master2.his.net
172.24.78.23  harbor1.his.net
172.24.78.24  harbor2.his.net
172.24.78.25  etcd1.his.net
172.24.78.26  etcd2.his.net
172.24.78.27  etcd3.his.net
172.24.78.28  ha1.his.net
172.24.78.29  ha2.his.net
172.24.78.12  node1.his.net
172.24.78.31  node2.his.net

手动二进制部署K8S
ansible部署

apt-get update && apt-get install -y apt-transport-https
curl https://mirrors.aliyun.com/kubernetes/apt/doc/apt-key.gpg | apt-key add -
cat <<EOF >/etc/apt/sources.list.d/kubernetes.list
deb https://mirrors.aliyun.com/kubernetes/apt/ kubernetes-xenial main
EOF
apt-get update
apt-get upgrade

sudo add-apt-repository main
sudo add-apt-repository universe
sudo add-apt-repository restricted
sudo add-apt-repository multiverse
sudo apt-get update
apt-get install git apt-get install python2.7
ln -s /usr/bin/python2.7 /usr/bin/python

安装依赖软件

sudo apt-get install -y git
sudo apt-get install -y python2.7
ln -s /usr/bin/python2.7 /usr/bin/python
apt-get install git python-pip ansible -y

ssh同步公钥

apt-get install sshpass
ssh-keygen
回车3次

修改自动缩进

vim ~/.vimrc
set paste

修改hosts映射或在DNS中配置

vim /etc/hosts
172.24.78.23 harbor.his.net
172.24.78.24 harbor.his.net
172.24.78.21  master1.his.net
172.24.78.22  master2.his.net
172.24.78.23  harbor1.his.net
172.24.78.24  harbor2.his.net
172.24.78.25  etcd1.his.net
172.24.78.26  etcd2.his.net
172.24.78.27  etcd3.his.net
172.24.78.28  ha1.his.net
172.24.78.29  ha2.his.net
172.24.78.30  node1.his.net
172.24.78.31  node2.his.net

分发公钥脚本(master-12)

vim scp.sh
:set paste

#!/bin/bash
#目标主机列表
IP="
172.24.78.21
172.24.78.22
172.24.78.23
172.24.78.24
172.24.78.25
172.24.78.26
172.24.78.27
172.24.78.28
172.24.78.29
172.24.78.30
172.24.78.31
" 
for node in ${IP};do
    sshpass -p silence ssh-copy-id ${node} -o StrictHostKeyChecking=no
    if [ $? -eq 0 ];then
        echo "${node} 秘钥copy完成"
    else
        echo "${node} 秘钥copy失败"
    fi
done

#同步docker证书脚本

vim cer.sh

#!/bin/bash
#目标主机列表
IP="
172.24.78.21
172.24.78.22
172.24.78.23
172.24.78.24
172.24.78.25
172.24.78.26
172.24.78.27
172.24.78.28
172.24.78.29
172.24.78.30
172.24.78.31
" 
for node in ${IP};do
    sshpass -p silence ssh-copy-id ${node} -o StrictHostKeyChecking=no
    if [ $? -eq 0 ];then
        echo "${node} 秘钥copy完成"
        echo "${node} 秘钥copy完成,准备环境初始化....."
        ssh ${node} "mkdir /etc/docker/certs.d/harbor.his.net -p"
        echo "Harbor 证书目录创建成功!"
        scp /etc/docker/certs.d/harbor.his.net/harborca.crt ${node}:/etc/docker/certs.d/harbor.his.net/harborca.crt
        echo "Harbor 证书拷贝成功!"
        scp /etc/hosts ${node}:/etc/hosts
        echo "host 文件拷贝完成"
        scp -r /root/.docker ${node}:/root/
        echo "Harbor 认证文件拷贝完成!"
        scp -r /etc/resolv.conf ${node}:/etc/
    else
        echo "${node} 秘钥copy失败"
    fi
done

执行脚本同步

bash scp.sh
bash cer.sh

clone项目-master1-2

git clone -b 0.6.1 https://github.com/easzlab/kubeasz.git

###备份文件

mv /etc/ansible/* /opt/
mv kubeasz/* /etc/ansible/
cd /etc/ansible/
cp example/hosts.m-masters.example ./hosts

准备hosts文件

cd /etc/ansible
cp example/hosts.m-masters.example ./hosts
vim hosts

# 集群部署节点：一般为运行ansible 脚本的节点
# 变量 NTP_ENABLED (=yes/no) 设置集群是否安装 chrony 时间同步
[deploy]
172.24.78.21 NTP_ENABLED=no
# etcd集群请提供如下NODE_NAME，注意etcd集群必须是1,3,5,7...奇数个节点
[etcd]
172.24.78.25 NODE_NAME=etcd1
172.24.78.26 NODE_NAME=etcd2
172.24.78.27 NODE_NAME=etcd3
[new-etcd] # 预留组，后续添加etcd节点使用
#172.24.78.x NODE_NAME=etcdx
[kube-master]
172.24.78.21
[new-master] # 预留组，后续添加master节点使用
#172.24.78.22
[kube-node]
172.24.78.30
[new-node] # 预留组，后续添加node节点使用
#172.24.78.31
# 参数 NEW_INSTALL：yes表示新建，no表示使用已有harbor服务器
# 如果不使用域名，可以设置 HARBOR_DOMAIN=""
[harbor]
#172.24.78.23 HARBOR_DOMAIN="harbor.his.net" NEW_INSTALL=no
# 负载均衡(目前已支持多于2节点，一般2节点就够了) 安装 haproxy+keepalived
[lb]
172.24.78.29 LB_ROLE=backup
172.24.78.28 LB_ROLE=master
#【可选】外部负载均衡，用于自有环境负载转发 NodePort 暴露的服务等
[ex-lb]
#172.24.78.6 LB_ROLE=backup EX_VIP=172.24.78.250
#172.24.78.7 LB_ROLE=master EX_VIP=172.24.78.250
[all:vars]
# ---------集群主要参数---------------
#集群部署模式：allinone, single-master, multi-master
DEPLOY_MODE=multi-master
#集群主版本号，目前支持: v1.8, v1.9, v1.10，v1.11, v1.12, v1.13
K8S_VER="v1.13"
# 集群 MASTER IP即 LB节点VIP地址，为区别与默认apiserver端口，设置VIP监听的服务端口8443
# 公有云上请使用云负载均衡内网地址和监听端口
MASTER_IP="172.24.78.18"
KUBE_APISERVER="https://{{ MASTER_IP }}:6443"
# 集群网络插件，目前支持calico, flannel, kube-router, cilium
CLUSTER_NETWORK="calico"
# 服务网段 (Service CIDR），注意不要与内网已有网段冲突
SERVICE_CIDR="10.20.0.0/16"
# POD 网段 (Cluster CIDR），注意不要与内网已有网段冲突
CLUSTER_CIDR="172.31.0.0/16"
# 服务端口范围 (NodePort Range)
NODE_PORT_RANGE="20000-60000"
# kubernetes 服务 IP (预分配，一般是 SERVICE_CIDR 中第一个IP)
CLUSTER_KUBERNETES_SVC_IP="10.20.0.1"
# 集群 DNS 服务 IP (从 SERVICE_CIDR 中预分配)
CLUSTER_DNS_SVC_IP="10.20.254.254"
# 集群 DNS 域名
CLUSTER_DNS_DOMAIN="his20.local."
# 集群basic auth 使用的用户名和密码
BASIC_AUTH_USER="admin"
BASIC_AUTH_PASS="silence"
# ---------附加参数--------------------
#默认二进制文件目录
bin_dir="/usr/bin"
#证书目录
ca_dir="/etc/kubernetes/ssl"
#部署目录，即 ansible 工作目录，建议不要修改
base_dir="/etc/ansible"

准备二进制文件

cd /etc/ansible/bin
tar xvf k8s.1-13-5.tar.gz
mv bin/* .

开始按步骤部署
环境初始化

cd /etc/ansible
ansible-playbook 01.prepare.yml

提示

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.21               : ok=33   changed=13   unreachable=0    failed=0   
172.24.78.25               : ok=15   changed=13   unreachable=0    failed=0   
172.24.78.26               : ok=15   changed=13   unreachable=0    failed=0   
172.24.78.27               : ok=15   changed=13   unreachable=0    failed=0   
172.24.78.28               : ok=29   changed=21   unreachable=0    failed=0   
172.24.78.29               : ok=29   changed=21   unreachable=0    failed=0   
172.24.78.30               : ok=15   changed=13   unreachable=0    failed=0

部署etcd集群

ansible-playbook 02.etcd.yml

提示

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.25               : ok=11   changed=9    unreachable=0    failed=0   
172.24.78.26               : ok=11   changed=9    unreachable=0    failed=0   
172.24.78.27               : ok=11   changed=9    unreachable=0    failed=0

各etcd服务器验证etcd服务（登录各etcd服务器）

定义etcd-IP变量

export NODE_IPS="172.24.78.25 172.24.78.26 172.24.78.27"

测试etcd服务

for ip in ${NODE_IPS}; do ETCDCTL_API=3 /usr/bin/etcdctl --endpoints=https://${ip}:2379 --cacert=/etc/kubernetes/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem endpoint health; done

提示

https://172.24.78.25:2379 is healthy: successfully committed proposal: took = 2.397926ms
https://172.24.78.26:2379 is healthy: successfully committed proposal: took = 2.771018ms
https://172.24.78.27:2379 is healthy: successfully committed proposal: took = 2.593279ms

部署docker

修改host文件，启用相关主机

vim hosts
[new-master] # 预留组，后续添加master节点使用
172.24.78.22
[kube-node]
172.24.78.30
[new-node] # 预留组，后续添加node节点使用
172.24.78.31

ansible-playbook 03.docker.yml

提示

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.22               : ok=13   changed=10   unreachable=0    failed=0   
172.24.78.30               : ok=12   changed=4    unreachable=0    failed=0   
172.24.78.31               : ok=12   changed=9    unreachable=0    failed=0

部署master

ansible-playbook 04.kube-master.yml

提示

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.21               : ok=39   changed=34   unreachable=0    failed=0

部署node

ansible-playbook 05.kube-node.yml

备注：node节点必须安装docker

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.30               : ok=24   changed=23   unreachable=0    failed=0

部署网络服务calico

ansible-playbook 06.network.yml

报错提示


fatal: [172.24.78.30 -> 172.24.78.21]: FAILED! => {"attempts": 15, "changed": true, "cmd": "/usr/bin/kubectl get pod -n kube-system -o wide|grep 'calico-node'|grep ' 172.24.78.30 '|awk '{print $3}'", "delta": "0:00:00.098717", "end": "2020-08-15 06:46:14.391950", "rc": 0, "start": "2020-08-15 06:46:14.293233", "stderr": "", "stderr_lines": [], "stdout": "Init:0/1", "stdout_lines": ["Init:0/1"]}

PLAY RECAP ************************************************************************************************************************************************************
172.24.78.21               : ok=16   changed=14   unreachable=0    failed=0   
172.24.78.30               : ok=7    changed=6    unreachable=0    failed=1

解决办法：下载镜像并上传至本地镜像仓库

vim roles/calico/defaults/main.yml
#更新支持calico
calica_ver: "3.3.2"

导入3.3.2镜像包

cd /opt
tar xvf calico-release-v3.3.2.tgz
cd release-v3.3.2/images
docker load -i calico-node.tar
docker tag calico/node:v3.3.2 harbor.his.net/baseimages/calico-node:v3.3.2
docker push harbor.his.net/baseimages/calico-node:v3.3.2
#导入镜像
docker load -i calico-cni.tar
docker tag calico/cni:v3.3.2 harbor.his.net/baseimages/calico-cni:v3.3.2
docker push harbor.his.net/baseimages/calico-cni:v3.3.2
#导入镜像
docker load -i calico-kube-controllers.tar
docker tag calico/kube-controllers:v3.3.2 harbor.his.net/baseimages/calico-kube-controllers:v3.3.2
docker push harbor.his.net/baseimages/calico-kube-controllers:v3.3.2

查看镜像是否上传成功

修改yaml文件为本地仓库地址

vim /etc/ansible/roles/calico/templates/calico-v3.3.yaml.j2
containers:
-name:calico-node
   image:harbor.his.net/baseimages/calico-node:v3.3.2
-name:calico-cni
   image:harbor.his.net/baseimages/calico-cni:v3.3.2
-name:calico-kube-controllers
   image:harbor.his.net/baseimages/calico-kube-controllers:v3.3.2

替换calicoctl文件版本

mv /etc/ansible/bin/calicoctl /tmp

将本地calicoctl上传至/etc/ansible/bin/calicoctl

chmod a+x /etc/ansible/bin/calicoctl
root@master1:/etc/ansible/bin# ./calicoctl version
Client Version:    v3.3.2
Build date:        2018-12-03T15:10:51+0000
Git commit:        594fd84e
Cluster Version:   v3.4.1
Cluster Type:      k8s,bgp

执行ansible

ansible-playbook 06.network.yml

提示

#异常
PLAY RECAP *********************************************************************************************************************************************
172.24.78.21               : ok=15   changed=8    unreachable=0    failed=0   
172.24.78.30               : ok=8    changed=4    unreachable=0    failed=0 
#正常
PLAY RECAP **********************************************************************************************************************************************
172.24.78.21               : ok=8    changed=2    unreachable=0    failed=0   
172.24.78.22               : ok=15   changed=7    unreachable=0    failed=0   
172.24.78.30               : ok=8    changed=3    unreachable=0    failed=0   
172.24.78.31               : ok=8    changed=3    unreachable=0    failed=0

在节点查看是否能下载calico镜像

calicoctl node status

root@master1:/etc/ansible/bin# calicoctl node status
Calico process is running.

IPv4 BGP status
+--------------+-------------------+-------+----------+-------------+
| PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
+--------------+-------------------+-------+----------+-------------+
| 172.24.78.30 | node-to-node mesh | up    | 01:17:32 | Established |
+--------------+-------------------+-------+----------+-------------+

IPv6 BGP status
No IPv6 peers found.

root@master1:/opt/release-v3.3.2/images# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5445d7cb7b-62smh   1/1     Running   0          2m56s
calico-node-n75j8                          2/2     Running   0          94m
calico-node-rcmtc                          2/2     Running   2          95m
calico-node-vkcjr                          2/2     Running   0          66m
calico-node-vm7w5                          2/2     Running   0          71m
kubernetes-dashboard-74f5cf8499-wxgt7      1/1     Running   0          2m56s

添加node节点

#去掉新加node注释
vim /etc/ansible/hosts
[new-node] # 预留组，后续添加node节点使用
172.24.78.31

ansible-playbook 20.addnode.yml

提示

PLAY RECAP *********************************************************************************************************************************************
172.24.78.21               : ok=2    changed=1    unreachable=0    failed=0   
172.24.78.31               : ok=67   changed=54   unreachable=0    failed=0

添加master节点

 #注释掉lb组
 vim /etc/ansible/hosts
 [lb]
 #172.24.78.29 LB_ROLE=backup
 #172.24.78.28 LB_ROLE=master

 ansible-playbook 21.addmaster.yml

提示

 PLAY RECAP *********************************************************************************************************************************************
 172.24.78.21               : ok=2    changed=1    unreachable=0    failed=0   
 172.24.78.22               : ok=82   changed=68   unreachable=0    failed=0

查看结果

 calicoctl node status

提示

 Calico process is running.

 IPv4 BGP status
 +--------------+-------------------+-------+----------+-------------+
 | PEER ADDRESS |     PEER TYPE     | STATE |  SINCE   |    INFO     |
 +--------------+-------------------+-------+----------+-------------+
 | 172.24.78.30 | node-to-node mesh | up    | 01:17:32 | Established |
 | 172.24.78.31 | node-to-node mesh | up    | 01:41:33 | Established |
 | 172.24.78.22 | node-to-node mesh | up    | 01:46:20 | Established |
 +--------------+-------------------+-------+----------+-------------+

 IPv6 BGP status
 No IPv6 peers found.

 root@master1:/etc/ansible/bin# kubectl get nodes

提示

 NAME           STATUS                          ROLES    AGE     VERSION
 172.24.78.21   Ready,SchedulingDisabled        master   43h     v1.13.5
 172.24.78.22   Ready,SchedulingDisabled        master   65s     v1.13.5
 172.24.78.30   Ready                           node     43h     v1.13.5
 172.24.78.31   Ready                           node     5m52s   v1.13.5

k8s应用环境

dashboard(1.10.1)

导入dashboard镜像并上传至本地harbor服务器

 cd /opt/
 tar xvf dashboard-yaml_image-1.10.1.tar.gz
 docker load -i kubernetes-dashboard-amd64-v1.10.1.tar.gz
 docker tag gcr.io/google-containers/kubernetes-dashboard-amd64:v1.10.1 harbor.his.net/baseimages/kubernetes-dashboard-amd64:v1.10.1
 docker push harbor.his.net/baseimages/kubernetes-dashboard-amd64:v1.10.1

修改yaml文件中的dashboard镜像地址为本地harbor地址

 mv /etc/ansible/manifests/dashboard/* /tmp/
 tar xvf dashboard-yaml_image-1.10.1.tar.gz -C /etc/ansible/manifests/dashboard/
 cd /etc/ansible/manifests/dashboard/
 vim kubernetes-dashboard.yaml
 image: harbor.his.net/baseimages/kubernetes-dashboard-amd64:v1.10.1

创建服务

 kubectl apply -f .

提示

serviceaccount/admin-user created
clusterrolebinding.rbac.authorization.k8s.io/admin-user created
secret/kubernetes-dashboard-certs created
serviceaccount/kubernetes-dashboard created
role.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
rolebinding.rbac.authorization.k8s.io/kubernetes-dashboard-minimal created
deployment.apps/kubernetes-dashboard created
service/kubernetes-dashboard created
serviceaccount/dashboard-read-user created
clusterrolebinding.rbac.authorization.k8s.io/dashboard-read-binding created
clusterrole.rbac.authorization.k8s.io/dashboard-read-clusterrole created
clusterrole.rbac.authorization.k8s.io/ui-admin created
rolebinding.rbac.authorization.k8s.io/ui-admin-binding created
clusterrole.rbac.authorization.k8s.io/ui-read created
rolebinding.rbac.authorization.k8s.io/ui-read-binding created

验证dashboard启动完成

kubectl get pods -n kube-system

提示

root@master1:/opt/release-v3.3.2/images# kubectl get pods -n kube-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-5445d7cb7b-62smh   1/1     Running   0          2m56s
calico-node-n75j8                          2/2     Running   0          94m
calico-node-rcmtc                          2/2     Running   2          95m
calico-node-vkcjr                          2/2     Running   0          66m
calico-node-vm7w5                          2/2     Running   0          71m
kubernetes-dashboard-74f5cf8499-wxgt7      1/1     Running   0          2m56s

kubectl get svc -n kube-system

提示

NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.20.234.4   <none>        443:57125/TCP   43m

kubectl get svc -n kube-system

提示

NAME                   TYPE       CLUSTER-IP    EXTERNAL-IP   PORT(S)         AGE
kubernetes-dashboard   NodePort   10.20.234.4   <none>        443:57125/TCP   44m

kubectl cluster-info

提示

Kubernetes master is running at https://172.24.78.18:6443
kubernetes-dashboard is running at https://172.24.78.18:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

查看dashboard登录用户及密码

vim /etc/ansible/hosts
# 集群basic auth 使用的用户名和密码
BASIC_AUTH_USER="admin"
BASIC_AUTH_PASS="silence"
#输入三次

测试访问

https://172.24.78.18:6443/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy

令牌

token登录dashboard

kubectl -n kube-system get secret | grep admin-user
#admin-user-token-vlt4j                kubernetes.io/service-account-token   3      59m

kubectl -n kube-system describe secret admin-user-token-vlt4j

token:

eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJhZG1pbi11c2VyLXRva2VuLXZsdDRqIiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9zZXJ2aWNlLWFjY291bnQubmFtZSI6ImFkbWluLXVzZXIiLCJrdWJlcm5ldGVzLmlvL3NlcnZpY2VhY2NvdW50L3NlcnZpY2UtYWNjb3VudC51aWQiOiJjNTcyYTMwOC1lMDJlLTExZWEtOWJlNi1mZWZjZmUyNjMzYjQiLCJzdWIiOiJzeXN0ZW06c2VydmljZWFjY291bnQ6a3ViZS1zeXN0ZW06YWRtaW4tdXNlciJ9.kJhzTn58yTYu6BGvT55hH8C_W92xK85mRjD0vigqRUSKfUwxPbe0bnt47wJ0j6qvDRO-siofrwnQm9wvX0ge9yKBrXmcsNmiuocmMqK9jUjWUnXNEVIge3IVtgdYxEoETpiM_MSEE1sEHUh1sXv9V51--ObO43V7AQnyOXEQKEX4Tf6qv66q7YYuYHBWdBODyEWgBfRITq2tBA6lGefaQOR3x69WoQdx40yzejUXwuqzGc0bEy87w79372jDxSfWDdlAKha5FiXaSFd9IVI7YIjnbEt2VuvNJ_zc0PbZ2CbavDRbMyY3HHEw8s0jLOyeFzZsi-fVKWyRmQrEpFdinQ

制作Kubeconfig文件

设置token登录会话保持时间

vim /etc/ansible/manifests/dashboard/kubernetes-dashboard.yaml
      containers:
      - name: kubernetes-dashboard
        #image: k8s.gcr.io/kubernetes-dashboard-amd64:v1.8.3
        image: harbor.his.net/baseimages/kubernetes-dashboard-amd64:v1.10.1
        ports:
        - containerPort: 8443
          protocol: TCP
        args:
          - --auto-generate-certificates
          - --token-ttl=43200

部署kube-dns

导入镜像并上传至本地harbor

 mkdir -p /etc/ansible/manifests/dns
 docker load -i k8s-dns-kube-dns-amd64_1.14.13.tar.gz
 docker tag gcr.io/google-containers/k8s-dns-kube-dns-amd64:1.14.13 harbor.his.net/baseimages/k8s-dns-kube-dns-amd64:1.14.13
 docker push harbor.his.net/baseimages/k8s-dns-kube-dns-amd64:1.14.13

 docker load -i k8s-dns-sidecar-amd64_1.14.13.tar.gz
 docker tag gcr.io/google-containers/k8s-dns-sidecar-amd64:1.14.13 harbor.his.net/baseimages/k8s-dns-sidecar-amd64:1.14.13
 docker push harbor.his.net/baseimages/k8s-dns-sidecar-amd64:1.14.13

 docker load -i k8s-dns-dnsmasq-nanny-amd64_1.14.13.tar.gz
 docker tag gcr.io/google-containers/k8s-dns-dnsmasq-nanny-amd64:1.14.13 harbor.his.net/baseimages/k8s-dns-dnsmasq-nanny-amd64:1.14.13
 docker push harbor.his.net/baseimages/k8s-dns-dnsmasq-nanny-amd64:1.14.13

修改yaml文件中的镜像地址为本地harbor地址

vim kube-dns.yaml
      - name: kubedns
        image: harbor.his.net/baseimages/k8s-dns-kube-dns-amd64:1.14.13
        args:
        - --domain=his.local.

- name: dnsmasq
        image: harbor.his.net/baseimages/k8s-dns-dnsmasq-nanny-amd64:1.14.13

      - name: sidecar
        image: harbor.his.net/baseimages/k8s-dns-sidecar-amd64:1.14.13
        args:
        - --v=2
        - --logtostderr
        - --probe=kubedns,127.0.0.1:10053,kubernetes.default.svc.his.local,5,SRV
        - --probe=dnsmasq,127.0.0.1:53,kubernetes.default.svc.his.local,5,SRV

创建服务

kubectl apply -f kube-dns.yaml

提示

service/kube-dns created
serviceaccount/kube-dns created
configmap/kube-dns created
deployment.extensions/kube-dns created

监控组件heapster

heapster：数据采集
influxdb：数据存储
grafana：web展示

导入相应的镜像

 cp -r /etc/ansible/manifests/heapster /opt/
 cd /etc/ansible/manifests/heapster/
 rm -rf ./*
 docker load -i docker load -i heapster-amd64_v1.5.1.tar
 docker tag gcr.io/google-containers/heapster-amd64:v1.5.1 harbor.his.net/baseimages/heapster-amd64:v1.5.1
 docker push harbor.his.net/baseimages/heapster-amd64:v1.5.1

 docker load -i eapster-grafana-amd64-v4.4.3.tar
 docker tag 8cb3de219af7 harbor.his.net/baseimages/eapster-grafana-amd64:v4.4.3
 docker push harbor.his.net/baseimages/eapster-grafana-amd64:v4.4.3

 docker load -i heapster-influxdb-amd64_v1.3.3.tar 
 docker tag gcr.io/google-containers/heapster-influxdb-amd64:v1.3.3 harbor.his.net/baseimages/heapster-influxdb-amd64:v1.3.3
 docker push harbor.his.net/baseimages/heapster-influxdb-amd64:v1.3.3

更改yaml中的镜像地址

 vim heapster.yaml 
       containers:
       - name: heapster
         image: harbor.his.net/baseimages/heapster-amd64:v1.5.1

 vim grafana.yaml 
       containers:
       - name: grafana
         image: harbor.his.net/baseimages/eapster-grafana-amd64:v4.4.3

   vim influxdb.yaml 
     containers:
       - name: influxdb
         image: harbor.his.net/baseimages/heapster-influxdb-amd64:v1.3.3

创建服务

 kubectl apply -f .

提示

 service/monitoring-grafana created
 serviceaccount/heapster created
 clusterrolebinding.rbac.authorization.k8s.io/heapster created
 deployment.extensions/heapster created
 service/heapster created
 deployment.extensions/monitoring-influxdb created
 service/monitoring-influxdb created

来源：oschina

链接：https://my.oschina.net/u/4418449/blog/4503344

标签

etcd

Harbor

YAML

Ansible创建K8S集群kube

服务器信息

主机信息

软件信息

手动二进制部署K8S

ansible部署

ssh同步公钥

clone项目-master1-2

开始按步骤部署

环境初始化

部署etcd集群

部署docker

部署master

部署node

部署网络服务calico

添加node节点

添加master节点

k8s应用环境

dashboard(1.10.1)

部署kube-dns

监控组件heapster