1. 技术文章
  2. Ansible创建K8S集群环境

Ansible创建K8S集群环境

由 那年仲夏 提交于 2020-08-18 12:01:20

服务器信息

类型 服务器IP地址 备注
Ansible(2台) 172.24.78.21/22 K8S集群部署服务器，可以和在一起，需要配置在负载均衡上实现反向代理，dashboard的端口为8443
K8S Master(2台) 172.24.78.21/22 K8s控制端，通过一个VIP做主备高可用
Harbor(2台) 172.24.78.23/24 高可用镜像服务器
Etcd(最少3台) 172.24.78.25/26/27 保存k8s集群数据的服务器
Hproxy(2台) 172.24.78.28/29 高可用etcd代理服务器
Node节点(2-N台) 172.24.78.31/32/xxx 真正运行容器的服务器，高可用环境至少两台

Ansible创建K8S集群环境

主机信息

序号 类型 服务器IP 主机名 VIP
1 K8S Master1 172.24.78.21 master1.his.net 172.24.78.18
2 K8S Master2 172.24.78.22 master2.his.net 172.24.78.18
3 Harbor1 172.24.78.23 harbor1.his.net
4 Harbor2 172.24.78.24 harbor2.his.net
5 etcd节点1 172.24.78.25 etcd1.his.net
6 etcd节点2 172.24.78.26 etcd2.his.net
7 etcd节点3 172.24.78.27 etcd3.his.net
8 Haproxy1 172.24.78.28 ha1.his.net
9 Haproxy2 172.24.78.29 ha2.his.net
10 Node节点1 172.24.78.12 node1.his.net
3 Node节点2 172.24.78.31 node2.his.net

软件信息

端口：192.168.7.248：6443 #需要配置在负载均衡上实现反向代理，dashboard的端口为8443
操作系统：ubuntu server 1804
k8s版本： 1.13.5
calico：3.4.4


修改主机名及IP地址

vim /etc/netplan/50-cloud-init.yaml
            dhcp4: no
            dhcp6: no
            addresses: [172.24.78.25/25]
            gateway4: 172.24.78.1
            nameservers:
                    addresses: [34.34.34.34,202.96.134.133]
hostnamectl set-hostname master1.his.net...

添加hosts映射

172.24.78.21  master1.his.net
172.24.78.22  master2.his.net
172.24.78.23  harbor1.his.net
172.24.78.24  harbor2.his.net
172.24.78.25  etcd1.his.net
172.24.78.26  etcd2.his.net
172.24.78.27  etcd3.his.net
172.24.78.28  ha1.his.net
172.24.78.29  ha2.his.net
172.24.78.12  node1.his.net
172.24.78.31  node2.his.net

  • Keepalive集群配置

keepalived安装配置于ha1

apt-get install -y keepalived haproxy
cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf
vim /etc/keepalived/keepalived.conf
:%d
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state MASTER
    interface enp0s18
    virtual_router_id 1
    priority 100
    advert_int 3
    unicast_src_ip 172.24.78.28
    unicast_peer {
        172.24.78.29
    } 
    authentication {
        auth_type PASS
        auth_pass 123abc
    } 
    virtual_ipaddress {
        172.24.78.18 dev enp0s18 label enp0s18:1
    }
}

重启服务

systemctl restart keepalived

查看结果

root@ha1:~# ip addr
2: enp0s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fe:fc:fe:f7:b6:55 brd ff:ff:ff:ff:ff:ff
    inet 172.24.78.28/26 brd 172.24.78.63 scope global enp0s18
       valid_lft forever preferred_lft forever
    inet 172.24.78.18/32 scope global enp0s18:1
       valid_lft forever preferred_lft forever
    inet6 fe80::fcfc:feff:fef7:b655/64 scope link 
       valid_lft forever preferred_lft forever

keepalived安装配置于ha2

apt-get install -y keepalived haproxy
cp /usr/share/doc/keepalived/samples/keepalived.conf.vrrp /etc/keepalived/keepalived.conf
vim /etc/keepalived/keepalived.conf
:%d
! Configuration File for keepalived

global_defs {
   notification_email {
     acassen
   }
   notification_email_from Alexandre.Cassen@firewall.loc
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}

vrrp_instance VI_1 {
    state BACKUP
    interface enp0s18
    virtual_router_id 1
    priority 90
    advert_int 3
    unicast_src_ip 172.24.78.29
    unicast_peer {
        172.24.78.28
    } 
    authentication {
        auth_type PASS
        auth_pass 123abc
    } 
    virtual_ipaddress {
        172.24.78.18 dev enp0s18 label enp0s18:1
    }
}

重启服务

systemctl restart keepalived

查看结果

root@ha2:~# ip addr
2: enp0s18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether fe:fc:fe:ee:bf:0c brd ff:ff:ff:ff:ff:ff
    inet 172.24.78.29/26 brd 172.24.78.63 scope global enp0s18
       valid_lft forever preferred_lft forever
    inet 172.24.78.18/32 scope global enp0s18:1
       valid_lft forever preferred_lft forever
    inet6 fe80::fcfc:feff:feee:bf0c/64 scope link 
       valid_lft forever preferred_lft forever

  • Haproxy集群配置

Haproxy安装配置于ha1,ha2

vim /etc/haproxy/haproxy.cfg 
listen k8s_api_nodes_6443
    bind 172.24.78.18:6443
    mode tcp
    #balance leastconn
    server 172.24.78.21 172.24.78.21:6443 check inter 2000 fall 3 rise 5
    server 172.24.78.22 172.24.78.22:6443 check inter 2000 fall 3 rise 5

重启服务

systemctl restart haproxy

查看状态

root@ha1:~# ss -tnl
State         Recv-Q    Send-Q   Local Address:Port                   Peer Address:Port                
LISTEN       0         128       172.24.78.18:6443                     0.0.0.0:* 
root@ha2:~# ss -tnl
State        Recv-Q     Send-Q     Local Address:Port                                 Peer Address:Port                
LISTEN        0          128        172.24.78.18:6443                                      0.0.0.0:*

  • Harbor集群配置

更新并安装 Docker-CE

apt-get -y update && apt-get -y install docker-ce
systemctl start docker && systemctl enable docker
docker version

配置 docker 加速器

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
EOF
sudo systemctl daemon-reload
sudo systemctl restart docker

安装docker-compose及Harbor

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
chmod +x /usr/local/bin/docker-compose
tar xvf harbor-offline-installer-v1.7.5.tgz
ln -sv /usr/local/src/harbor /usr/local/

安装Harbor

证书制作

cd /usr/local/src/harbor
mkdir certs/
#生成私有key
openssl genrsa -out /usr/local/src/harbor/certs/harbor-ca.key
#签证
mkdir -p /root/.md
openssl req -x509 -new -nodes -key /usr/local/src/harbor/certs/harbor-ca.key -subj "/CN=harbor.his.net" -days 7120 -out /usr/local/src/harbor/certs/harborca.crt
vim harbor.cfg
hostname = harbor.his.net
ui_url_protocol = https
#修改证书位置
ssl_cert = /usr/local/src/harbor/certs/harbor-ca.crt
ssl_cert_key = /usr/local/src/harbor/certs/harbor-ca.key
#修改邮箱信息
email_server = smtp.163.com
email_server_port = 25
email_username = silencegan@163.com
email_password = ****
email_from = silencegan <silencegan@163.com>
email_ssl = false
email_insecure = false
#修改登录密码
harbor_admin_password = 123456
./install.sh

Ansible创建K8S集群环境

harbor重启命令

docker-compose stop
docker-compose up -d

测试登录

C:\Windows\System32\drivers\etc
172.24.78.23  harbor.his.net
https://harbor1.his.net/harbor/sign-in

admin s**

Ansible创建K8S集群环境

image-20200814160020058

client 同步在crt证书(Master1-2)

mkdir /etc/docker/certs.d/harbor.his.net -p

发送证书至master1-2(Harbor1-2)

scp /usr/local/src/harbor/certs/harborca.crt 172.24.78.21:/etc/docker/certs.d/harbor.his.net
scp /usr/local/src/harbor/certs/harborca.crt 172.24.78.22:/etc/docker/certs.d/harbor.his.net

添加host文件解析 (Master1-2)

vim /etc/hosts
172.24.78.23 harbor.his.net
172.24.78.24 harbor.his.net

Master(1-2)安装docker

apt-get -y update && apt-get -y install docker-ce
systemctl start docker && systemctl enable docker
docker version

配置 docker 加速器

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
"registry-mirrors": ["https://9916w1ow.mirror.aliyuncs.com"]
}
#手动输入
EOF

重启docker

systemctl daemon-reload
systemctl restart docker

测试登录harbor

docker login harbor.his.net

root@master1:/etc/docker/certs.d/harbor.his.net# docker login harbor.his.net
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store




Login Succeeded

root@master2:~# docker login harbor.his.net
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store




Login Succeeded

测试push镜像到harbor

docker pull alpine
docker tag alpine harbor.his.net/library/alpine:2020
docker push harbor.his.net/library/alpine:2020

The push refers to repository [harbor.his.net/library/alpine]
50644c29ef5a: Pushed
2020: digest: sha256:a15790640a6690aa1730c38cf0a440e2aa44aaca9b0e8931a9f2b0d7cc90fd65 size: 528

Ansible创建K8S集群环境

标签
Harbor
Docker
etcd
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!