k8s部署集群、自定义仓库、上线部署项目

安装k8s+docker集群环境（基于centos7系统）

集群机器：
centos-master = 192.168.121.9
centos-minion-1 = 192.168.121.65
centos-minion-2 = 192.168.121.66
centos-minion-3 = 192.168.121.67

1.配置yum源
[centos-master]:cat /etc/yum.repos.d/virt7-docker-common-release.repo

[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0

加载安装包：
[centos-master]:yum repolist

virt7-docker-common-release virt7-docker-common-release 15

2.安装集群必要软件—-etcd/flannel/kubernetes
Etcd服务在k8s集群中用于配置共享和服务发现。

Flannel是针对k8s设计一个网络规划服务，让集群中的不同节点主机创建的Docker容器都具有全集群唯一的虚拟IP地址。

[centos-master]:yum -y install –enablerepo=virt7-docker-common-release kubernetes etcd flannel

3.如果集群中没有使用DNS解析，那么需要在master节点的/etc/hosts中添加node的主机名信息，比如：

echo "192.168.121.9    centos-master
192.168.121.65    centos-minion-1
192.168.121.66  centos-minion-2
192.168.121.67  centos-minion-3" >> /etc/hosts

4.修改配置master节点的kubernetes配置文件/etc/kubernetes/config
#表示错误日志记录到文件还是输出到stderr
KUBE_LOGTOSTDERR="–logtostderr=true"

#日志等级
KUBE_LOG_LEVEL="--v=0"

#允许运行特权容器
KUBE_ALLOW_PRIV="--allow-privileged=false"

#apiserver的服务地址，controller-manager、scheduler及kubelet都会用到这个配置，这里配置为192.168.121.9
KUBE_MASTER="--master=http://192.168.121.9:8080"

5.k8s集群中涉及的端口比较多，所以centos中的防火墙需要设置对应规则，并需关闭selinux。为确保k8s集群的正常运行，我们可以直接关闭iptables与seliinux服务。

setenforce 0
systemctl stop firewalld.service
systemctl stop firewalld.service

6.修改配置master节点的etcd配置文件/etc/etcd/etcd.conf
etcd服务的可调参数比较多，根据需求开启对应功能，此处我们大概调整如下几个功能：

# [member]
#etcd名称
ETCD_NAME=default

#etcd数据存储位置
ETCD_DATA_DIR="/var/lib/etcd/default.etcd"

#监听的端口
ETCD_LISTEN_CLIENT_URLS="http://0.0.0.0:2379"

#[cluster]
#集群监听的端口
ETCD_ADVERTISE_CLIENT_URLS="http://0.0.0.0:2379"

7.修改配置master节点的apiserver配置文件/etc/kubernetes/apiserver

#监听的接口，如果配置为127.0.0.1则只监听localhost，配置为0.0.0.0会监听所有接口，这里配置为0.0.0.0
KUBE_API_ADDRESS="--address=0.0.0.0"

#apiserver的监听端口，默认8080
KUBE_API_PORT="--port=8080"

#minion上kubelet监听的端口，默认10250
KUBELET_PORT="--kubelet-port=10250"

#etcd服务地址，前面已经启动了etcd服务,端口为2379
KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:2379"

#kubernetes可以分配的ip的范围，kubernetes启动的每一个pod以及serveice都会分配一个ip地址，将从这个范围分配
KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"

#需要额外添加的配置项，简单地启用一个集群无需配置
KUBE_API_ARGS=""

8.启动并且配置etcd的网段，此网段一定是未被使用的

systemctl start etcd
etcdctl mkdir /kube-centos/network
etcdctl mk /kube-centos/network/config "{ \"Network\": \"172.30.0.0/16\", \"SubnetLen\": 24, \"Backend\": { \"Type\": \"vxlan\" } }"

9.修改配置master节点的flanneld配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口
FLANNEL_ETCD_ENDPOINTS="http://192.168.121.9:2379"

#服务范围
FLANNEL_ETCD_PREFIX="/kube-centos/network"

#其他
FLANNEL_OPTIONS=""

10.启动k8s集群

for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler flanneld; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

以上就是k8s，master节点的安装及配置
接下来，配置nodes节点

11.修改配置nodes节点kubelet的配置文件/etc/kubernetes/kubelet

#minion监听的地址，每个minion根据实际的ip配置,这里配置为0.0.0.0
KUBELET_ADDRESS="--address=0.0.0.0"

#监听的端口
KUBELET_PORT="--port=10250"

#apiserver的访问地址及端口
KUBELET_API_SERVER="--api-servers=http://192.168.121.9:8080"

#额外增加的参数
KUBELET_ARGS="--logtostderr=false --v=0 --log-dir=/data/logs/kubernetes"

12.修改配置nodes节点flanneld的配置文件/etc/sysconfig/flanneld

#etcd的访问地址及端口
FLANNEL_ETCD="http://192.168.121.9:2379"

#etcd服务范围
FLANNEL_ETCD_KEY="/kube-centos/network"

13.启动k8s集群服务

for SERVICES in kube-proxy kubelet flanneld docker; do
    systemctl restart $SERVICES
    systemctl enable $SERVICES
    systemctl status $SERVICES
done

14.设置kubectl的配置文件

kubectl config set-cluster default-cluster --server=http://192.168.121.9:8080
kubectl config set-context default-context --cluster=default-cluster --user=default-admin
kubectl config use-context default-context

15.检查集群状态
[centos-master]:kubectl get nodes

NAME                   STATUS     AGE     VERSION
centos-minion-1        Ready      3d      v1.5.0+fff5156
centos-minion-2        Ready      3d      v1.5.0+fff5156
centos-minion-3        Ready      3d      v1.5.0+fff5156

至此，集群构建完毕

搭建私有库
私有库用于系统内部存储成品镜像，能够快速进行下载及被k8s调度。

1.下载并启动私有库

[centos-master]:docker run --name registry -v /etc/localtime:/etc/localtime -v /opt/registry:/var/lib/registry -p 5000:5000 -itd docker.io/registry

#--name 表示启动的容器后名称，此处为registry
#-v 表示挂载路径  格式为宿主机路径:容器内路径
#-p 表示映射端口  格式为宿主机端口:容器内端口
#-itd   docker的内部参数，此处声明后台运行容器并分配一个伪终端并绑定到容器的标准输入上，后跟镜像名称此处为docker.io/registry

2.创建一个secret服务，用于k8s调度私有库容器时的“令牌”。简单来说，secret服务就是一个存储密码的服务

[centos-master]:kubectl create secret docker-registry registrykey --docker-server=registry.evehicle.cn --docker-username=docker --docker-password=docker --docker-email=lienhua@zhongchuangsanyou.com

[centos-master]:kubectl get secret
NAME          TYPE                      DATA      AGE
registrykey   kubernetes.io/dockercfg   1         6s

此时登录时会提示认证错误

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn
Flag --email has been deprecated, will be removed in 1.13.
Error response from daemon: login attempt to https://registry.evehicle.cn/v2/ failed with status: 401 Unauthorized

这是因为Docker官方是推荐采用Secure Registry的工作模式的，即transport采用tls。这样我们就需要为Registry配置tls所需的key和crt文件了

3.配置nginx反向代理
[centos-master]: cat registry.evehicle.cn.conf

# For versions of nginx > 1.3.9 that include chunked transfer encoding support
# Replace with appropriate values where necessary

upstream docker-registry {
  server 192.168.121.9:5000;
  #server 10.44.170.95:5000;
}

# uncomment if you want a 301 redirect for users attempting to connect
# on port 80
# NOTE: docker client will still fail. This is just for convenience
# server {
#   listen *:80;
#   server_name my.docker.registry.com;
#   return 301 https://$server_name$request_uri;
# }

server {
    listen 443;
    server_name registry.evehicle.cn;

    ssl on;
    ssl_certificate ssl/registry.evehicle.cn.crt;
    ssl_certificate_key ssl/registry.evehicle.cn.key;

    client_max_body_size 0; # disable any limits to avoid HTTP 413 for large image uploads

    # required to avoid HTTP 411: see Issue #1486 (https://github.com/docker/docker/issues/1486)
    chunked_transfer_encoding on;

    location / {
        auth_basic  "Restricted";
        auth_basic_user_file  passwd;
        add_header 'Docker-Distribution-Api-Version' 'registry/2.0' always;

        proxy_pass                          http://docker-registry;
        proxy_set_header  Host              $http_host;   # required for docker client's sake
        proxy_set_header  X-Real-IP         $remote_addr; # pass on real client's IP
        proxy_set_header  X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;
        proxy_read_timeout                  900;
        }

    location /_ping {
        auth_basic off;
        include               docker-registry.conf;
    }

    location /v1/_ping {
        auth_basic off;
        include               docker-registry.conf;
    }

    location /v2/_ping {
        auth_basic off;
        include               docker-registry.conf;
    }
}

将key及crt证书文件放到…/ssl目录下。使用htpasswd生成密码放于./上一级目录

 htpasswd -bcm passwd docker docker
 #-c：创建一个加密文件
 #-m：md5加密，默认可不填写
 #-b：表示用户名密码在命令行中一并输入，不用分别填写

4.再次登录

[centos-master]:docker login -u docker -p docker -e lienhua@zhongchuangsanyou.com registry.evehicle.cn
	
Login Succeeded
表示成功，此时再pull\push既在私有库中进行

构建服务
docker的本意是将代码包含在容器内制作成镜像形成“产品”。但出于公司的（频繁修改代码及服务器资源受限）的特殊性，我们将代码以“外挂”的形式运行在宿主机上。下面以部署官网（apache）服务为例：
1.从docker的公有库里下载centos7的原生镜像

[centos-master]:docker pull centos

Using default tag: latest
Trying to pull repository docker.io/library/centos ...
latest: Pulling from docker.io/library/centos
d9aaf4d82f24: Downloading [>              ]   540 kB/73.39 MB
d9aaf4d82f24: Pulling fs layer
Digest: sha256:eba772bac22c86d7d6e72421b4700c3f894ab6e35475a34014ff8de74c10872e
Status: Downloaded newer image for centos:latest

2.编写Dockerfile制造apache基础镜像

######httpd####
FROM centos
MAINTAINER lienhua lienhua@zhongchuangsanyou.com
RUN yum -y install epel-release
RUN yum -y install httpd  php php-mysql php-memcache* php-mbstring
ADD httpd.conf /etc/httpd/conf/httpd.conf

EXPOSE 80

CMD ["/usr/sbin/apachectl", "-D", "FOREGROUND"]

其中httpd.conf文件需要在当前目录下真实存在，此处其内容为

ServerRoot "/etc/httpd"
Listen 80
Listen 8080
Include conf.modules.d/*.conf
Include zcsy/*.conf
User apache
Group apache
ServerAdmin root@localhost
<Directory />
    AllowOverride none
    Require all denied
</Directory>
DocumentRoot "/var/www/html"
<Directory "/var/www">
    AllowOverride None
    Require all granted
</Directory>
<Directory "/var/www/html">
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>
<IfModule dir_module>
    DirectoryIndex index.html
</IfModule>
<Files ".ht*">
    Require all denied
</Files>
ErrorLog "logs/error_log"
LogLevel warn
<IfModule log_config_module>
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common
    <IfModule logio_module>
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>
    CustomLog "logs/access_log" combined
</IfModule>
<IfModule alias_module>
    ScriptAlias /cgi-bin/ "/var/www/cgi-bin/"
</IfModule>
<Directory "/var/www/cgi-bin">
    AllowOverride None
    Options None
    Require all granted
</Directory>
<IfModule mime_module>
    TypesConfig /etc/mime.types
    AddType application/x-compress .Z
    AddType application/x-gzip .gz .tgz
    AddType application/x-httpd-php .php
    AddType text/html .shtml
    AddOutputFilter INCLUDES .shtml
</IfModule>
AddDefaultCharset UTF-8
<IfModule mime_magic_module>
    MIMEMagicFile conf/magic
</IfModule>
EnableSendfile off
EnableMMAP off
IncludeOptional conf.d/*.conf

执行[centos-master]:docker build -t registry.evehicle.cn/httpd . 命令制作名为”registry.evehicle.cn/httpd”的镜像（注意此处的点必须要有，并且其意义代表当前目录下的Dockerfile文件）

3.将制作好的镜像上传到私有库

docker push registry.evehicle.cn/httpd

4.编写启动apache服务的yaml文件

[centos-master]:cat 13-rc-httpd.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: 13-rc-httpd
  labels:
    name: 13-rc-httpd
spec:
  replicas: 2
  selector:
    name: 13-rc-httpd
  template:
    metadata:
      labels:
        name: 13-rc-httpd
    spec:
      containers:
      - name: 13-rc-httpd
        image: registry.evehicle.cn/httpd
        env:
        - name: LANG
          value: en_US.UTF-8
        ports:
        - containerPort: 80
          hostPort: 80
        volumeMounts:
        - name: time
          mountPath: /etc/localtime
        - name: zcsy
          mountPath: /etc/httpd/zcsy
        - name: deploy
          mountPath: /docker/httpd/deploy
        - name: log
          mountPath: /var/log/httpd
      volumes:
        - name: time
          hostPath:
            path: /etc/localtime
        - name: zcsy
          hostPath:
            path: /docker/httpd/zcsy
        - name: deploy
          hostPath:
            path: /docker/httpd/deploy
        - name: log
          hostPath:
            path: /docker/httpd/log
      nodeSelector:
        slave: "13"
      imagePullSecrets:
      - name: registrykey

5.给其中一个node加上标签为“13”

kubectl label nodes centos-minion-1 slave=13

6.此时拥有标签“13”的nodes应具备的条件

/docker/httpd/zcsy下需要有官网的配置文件

<VirtualHost *:80>
   ServerName www.evehicle.cn
  DocumentRoot /var/deploy/wordpress/
        RewriteEngine on
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-d
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_FILENAME} !-f
        RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !^.*\.(ico|pdf|flv|jpe?g|js|gif|png|html|shtml|zip|xml|gz|rar|swf|txt|apk|bmp|css|m4a|ogg|mp3|ipa|plist)$
        RewriteCond %{REQUEST_URI} !^/server-status$
        RewriteRule . /index.php [QSA,PT,L]

</VirtualHost>
<Directory /var/deploy/wordpress/>
    Options FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

以及/docker/httpd/deploy下需要有官网的代码

7.运行yaml文件启动容器

[centos-master]: kuberctl create -f 13-rc-httpd.yaml

8.查看服务

[centos-master]: kuberctl get rc

NAME                 DESIRED   CURRENT   AGE
13-rc-httpd          2         2         168d

9.程序中涉及的mysql\redis\memcache等服务也需使用容器运行起来

[centos-master]: docker pull redis
[centos-master]: docker tag registry.evehicle.cn/redis redis
[centos-master]: docker push registry.evehicle.cn/redis
[centos-master]: kubectl create -f rc-redis.yaml
[centos-master]: cat rc-redis.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: redis
  labels:
    name: redis
spec:
  replicas: 2
  selector:
    name: redis
  template:
    metadata:
      labels:
        name: redis
    spec:
      containers:
      - name: redis
        image: registry.evehicle.cn/redis
        ports:
        - containerPort: 6379
          hostPort: 6379
        volumeMounts:
        - name: data
          mountPath: /data
        - name: time
          mountPath: /etc/localtime
      volumes:
        - name: data
          hostPath:
            path: /docker/redis/6379
        - name: time
          hostPath:
            path: /etc/localtime
      nodeSelector:
        slave: "13"
      imagePullSecrets:
      - name: registrykey

启动memcache
[centos-master]: docker pull memcache
[centos-master]: docker tag registry.evehicle.cn/memcached memcache
[centos-master]: docker push registry.evehicle.cn/memcached
[centos-master]: kubectl create -f rc-memcached.yaml
[centos-master]: cat rc-memcached.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: memcached
  labels:
    name: memcached
spec:
  replicas: 3
  selector:
    name: memcached
  template:
    metadata:
      labels:
        name: memcached
    spec:
      containers:
      - name: memcached
        image: registry.evehicle.cn/memcached
        ports:
        - containerPort: 11211
          hostPort: 11211
      #nodeSelector:
      #  slave: "13"
      imagePullSecrets:
      - name: registrykey

制造mysql镜像
[centos-master]: cat Dockerfile

FROM alpine


COPY startup.sh /startup.sh
RUN addgroup mysql && \
    adduser -H -D -s /bin/false -G mysql mysql && \
    apk add --update mysql mysql-client && rm -f /var/cache/apk/* && \
    mkdir /data && \
    chown -R mysql:mysql /data /etc/mysql && \
    chmod 755 /startup.sh \
    ;


WORKDIR /data
VOLUME /data
VOLUME /etc/mysql


EXPOSE 3306
CMD ["/startup.sh"]

启动mysql（建议mysql在宿主机启动）
[centos-master]: docker build -t registry.evehicle.cn/mysql
[centos-master]: docker push registry.evehicle.cn/mysql
[centos-master]: kubectl create -f rc-mysql.yaml
[centos-master]: cat rc-mysql.yaml

apiVersion: v1
kind: ReplicationController
metadata:
  name: 13-rc-mysql
  labels:
    name: 13-rc-mysql
spec:
  replicas: 2
  selector:
    name: 13-rc-mysql
  template:
    metadata:
      labels:
        name: 13-rc-mysql
    spec:
      containers:
      - name: 13-rc-mysql
        image: registry.evehicle.cn/mysql
        env:
        - name: MYSQL_DATABASE
          value: admin
        - name: MYSQL_USER
          value: tony
        - name: MYSQL_PASSWORD
          value: 456
        - name: MYSQL_ROOT_PASSWORD
          value: 123
        ports:
        - containerPort: 3306
          hostPort: 3306
        volumeMounts:
        - name: time
          mountPath: /etc/localtime
        - name: data
          mountPath: /data
        - name: etc
          mountPath: /etc/mysql
        - name: run
          mountPath: /run/mysqld
      volumes:
        - name: time
          hostPath:
            path: /etc/localtime
        - name: data
          hostPath:
            path: /docker/mysql/data
        - name: etc
          hostPath:
            path: /docker/mysql/etc
        - name: run
          hostPath:
            path: /docker/mysql/run
      nodeSelector:
        slave: "13"
      imagePullSecrets:
      - name: registrykey

为方便代码编写及统一管理，应提前做好内部DNS解析。将所负责的应用规整到对应的机器上。

来源：CSDN

作者：csdnYXJY

链接：https://blog.csdn.net/CSDNYXJY/article/details/103909987