一、前提
本次实践前,需已完成以下过程:
1、搭建好一个Kubernetes集群(本实践为单节点集群),网上参考较多,不赘述。
2、选取kubernetes集群外的一台服务器安装 NFS服务端,并在集群内每个节点安装 NFS客户端;
NFS服务端所在的服务器IP为 10.141.211.178
,记为 nfs server;而集群master服务器,记为 master;
(1) nfs server创建存储目录 /data/k8s/
并执行命令:chmod 755 /data/k8s/
, 并关闭防火墙
(2) nfs server安装NFS,执行: yum -y install nfs-utils rpcbind
,
再配置NFS,执行:vi /etc/exports
,在该文件内添加内容:/data/k8s *(rw,sync,no_root_squash)
,
然后启动NFS服务,执行:
# systemctl start rpcbind && systemctl enable rpcbind # systemctl start nfs && systemctl enable nfs
(3) master同样安装并启动NFS,执行:
# yum -y install nfs-utils rpcbind # systemctl start rpcbind && systemctl enable rpcbind # systemctl start nfs && systemctl enable nfs
再执行:showmount -e 10.141.211.178
,可看到共享目录 /data/k8s
二、集群安装Jenkins
Jenkins master的安装,需要将数据持久化。可以利用NFS作为存储资源,创建PVC对象来挂载。PV/PVC配置文件pvc.yaml如下:
apiVersion: v1 kind: PersistentVolume metadata: name: opspv spec: capacity: storage: 20Gi accessModes: - ReadWriteMany persistentVolumeReclaimPolicy: Delete nfs: server: 10.141.211.178 #注意:此处为NFS服务器的地址 path: /data/k8s --- kind: PersistentVolumeClaim apiVersion: v1 metadata: name: opspvc namespace: kube-ops spec: accessModes: - ReadWriteMany resources: requests: storage: 20Gi
同时,对于即将创建的Jenkins master资源对象,需要授予其一些权限,比如增删改查等。相应的配置文件rbac.yaml如下:
apiVersion: v1 kind: ServiceAccount metadata: name: jenkins namespace: kube-ops --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1beta1 metadata: name: jenkins rules: - apiGroups: ["extensions", "apps"] resources: ["deployments"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["services"] verbs: ["create", "delete", "get", "list", "watch", "patch", "update"] - apiGroups: [""] resources: ["pods"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create","delete","get","list","patch","update","watch"] - apiGroups: [""] resources: ["pods/log"] verbs: ["get","list","watch"] - apiGroups: [""] resources: ["secrets"] verbs: ["get"] --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: jenkins namespace: kube-ops roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: jenkins subjects: - kind: ServiceAccount name: jenkins namespace: kube-ops
然后,基于jenkins/jenkins:lts 镜像创建jenkins master镜像,配置文件jenkins.yaml 如下:
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: jenkins namespace: kube-ops spec: template: metadata: labels: app: jenkins spec: terminationGracePeriodSeconds: 10 serviceAccount: jenkins containers: - name: jenkins image: jenkins/jenkins:lts imagePullPolicy: IfNotPresent ports: - containerPort: 8080 name: web protocol: TCP - containerPort: 50000 name: agent protocol: TCP resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 500m memory: 512Mi livenessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 readinessProbe: httpGet: path: /login port: 8080 initialDelaySeconds: 60 timeoutSeconds: 5 failureThreshold: 12 volumeMounts: - name: jenkinshome subPath: jenkins mountPath: /var/jenkins_home env: - name: LIMITS_MEMORY valueFrom: resourceFieldRef: resource: limits.memory divisor: 1Mi - name: JAVA_OPTS value: -Xmx$(LIMITS_MEMORY)m -XshowSettings:vm -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85 -Duser.timezone=Asia/Shanghai securityContext: fsGroup: 1000 volumes: - name: jenkinshome persistentVolumeClaim: claimName: opspvc --- apiVersion: v1 kind: Service metadata: name: jenkins namespace: kube-ops labels: app: jenkins spec: selector: app: jenkins type: NodePort ports: - name: web port: 8080 targetPort: web nodePort: 30002 - name: agent port: 50000 targetPort: agent
最后,在一个目录内分别创建以上3个文件,执行命令如下:
# kubectl create namespace kube-ops # kubectl create -f pvc.yaml # kubectl create -f rbac.yaml # kubectl create -f jenkins.yaml(此步执行会出现文件权限问题,解决办法为: 先在nfs server服务器执行:chown -R 1000 /data/k8s/jenkins 然后在master执行:kubectl delete -f jenkins.yaml kubectl create -f jenkins.yaml )
此时,我们通过命令kubectl -n kube-ops get pod
可以查看到jenkins已成功创建。
三、Jenkins配置动态slave
1、初始化Jenkins配置
浏览器打开masterIP:30002
,如下:
其中的管理员密码,我们既可以进入容器内对应的目录查看,也可以在nfs server服务器上执行命令:cat /data/k8s/jenkins/secrets/initialAdminPassword
来查看;然后选择安装推荐的插件,如下:
然后添加管理员账户即可进入Jenkins界面。
2、配置jenkins slave
(1) 安装Kubernetes插件
进入 Manage Jenkins—>Manage Plugins—>可选插件(Available)—>Kubernetes plugin勾选,直接安装即可。
(2) 配置Kubernetes插件
点击Manage Jenkins—>Configure System—>云—>新增一个云—>Kubernetes,如下:
然后配置如下:
先注意 名称默认为kubernetes
,然后 Kubernetes地址 填写https://kubernetes.default.svc.cluster.local
,命名空间为kube-ops
;接着点击右边的 连接测试 按钮,如果显示Connection test successful
,表示Jenkins可以和Kubernetes集群正常通信了。最后,在Jenkins地址,填入:http://服务名.kube-ops.svc.cluster.local:8080
,如下所示:
(3) 配置 Kubernetes Pod Template
关于 Kubernetes Pod Template部分的配置,其实就是对jenkins slave的配置。具体配置如下:
图中标记的地方较为重要,不要填错。其中标签列表部分 后面仍有用到;Docker 镜像部分,是本人基于 cnych/jenkins:jnlp6 镜像基础上继续定制的镜像,包含maven、docker、docker-compose、kubectl等工具。
另外,添加卷如下:
添加这两个 Host Path Volume,是为了更好地在jenkins slave容器中使用docker 和 kubectl 工具,所以挂载了宿主机的部分目录。
然后,设置Service Account如下:
最后,点击 保存 即可。
3.测试jenkins slave
首先新建一个 名为test 的 Freestyle project 项目,其配置如下:
这里的标签表达式,正是Kubernetes Pod Template的标签列表的内容。
然后,增加构建步骤—>执行shell,如下:
具体shell如下:
echo "测试 Kubernetes 动态生成 jenkins slave" echo "===========mvn===========" mvn --version echo $PATH echo "==============docker in docker===========" which docker docker version echo "==============docker-compose===========" docker-compose version echo "=============kubectl=============" kubectl get pods
保存之后,点击 立即构建。控制台输出如下:
Started by user admin Running as SYSTEM Agent jnlp-slave-pk06f is provisioned from template Kubernetes Pod Template --- apiVersion: "v1" kind: "Pod" metadata: annotations: {} labels: jenkins: "slave" jenkins/jnlp-slave: "true" name: "jnlp-slave-pk06f" spec: containers: - env: - name: "JENKINS_SECRET" value: "********" - name: "JENKINS_AGENT_NAME" value: "jnlp-slave-pk06f" - name: "JENKINS_NAME" value: "jnlp-slave-pk06f" - name: "JENKINS_AGENT_WORKDIR" value: "/home/jenkins/agent" - name: "JENKINS_URL" value: "http://jenkins.kube-ops.svc.cluster.local:8080/" image: "zhongyuanzhao000/jenkins-slave:jnlp" imagePullPolicy: "IfNotPresent" name: "jnlp" resources: limits: {} requests: {} securityContext: privileged: false tty: true volumeMounts: - mountPath: "/var/run/docker.sock" name: "volume-0" readOnly: false - mountPath: "/root/.kube" name: "volume-1" readOnly: false - mountPath: "/home/jenkins/agent" name: "workspace-volume" readOnly: false workingDir: "/home/jenkins/agent" nodeSelector: {} restartPolicy: "Never" serviceAccount: "jenkins" volumes: - hostPath: path: "/var/run/docker.sock" name: "volume-0" - hostPath: path: "/root/.kube" name: "volume-1" - emptyDir: medium: "" name: "workspace-volume" Building remotely on jnlp-slave-pk06f (jnlp-slave) in workspace /home/jenkins/agent/workspace/test [test] $ /bin/sh -xe /tmp/jenkins3820575614440094591.sh + echo 测试 Kubernetes 动态生成 jenkins slave 测试 Kubernetes 动态生成 jenkins slave + echo ===========mvn=========== ===========mvn=========== + mvn --version Apache Maven 3.6.1 (d66c9c0b3152b2e69ee9bac180bb8fcc8e6af555; 2019-04-05T03:00:29+08:00) Maven home: /usr/local Java version: 1.8.0_212, vendor: Oracle Corporation, runtime: /usr/local/openjdk-8/jre Default locale: zh_CN, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix" + echo /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin /usr/local/openjdk-8/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin + echo ==============docker in docker=========== ==============docker in docker=========== + which docker /usr/local/bin/docker + docker version Client: Docker Engine - Community Version: 18.09.8 API version: 1.39 Go version: go1.10.8 Git commit: 0dd43dd87f Built: Wed Jul 17 17:38:58 2019 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 18.09.7 API version: 1.39 (minimum version 1.12) Go version: go1.10.8 Git commit: 2d0083d Built: Thu Jun 27 17:26:28 2019 OS/Arch: linux/amd64 Experimental: false + echo ==============docker-compose=========== ==============docker-compose=========== + docker-compose version docker-compose version 1.23.2, build 1110ad01 docker-py version: 3.6.0 CPython version: 3.6.7 OpenSSL version: OpenSSL 1.1.0f 25 May 2017 + echo =============kubectl============= =============kubectl============= + kubectl get pods NAME READY STATUS RESTARTS AGE jenkins-575b84fb7b-59s5h 1/1 Running 0 29h jnlp-slave-pk06f 1/1 Running 0 7s Finished: SUCCESS
其中,你可以发现jenkins创建了jnlp-slave-pk06f的slave对象;而当该任务执行完之后,你在master上获取pod就会发现 jnlp-slave-pk06f 这个slave自动消失了,这就是动态jenkins slave的简单体现。
四、参考
本次实践得益于诸多运维大神对于知识的不吝分享,十分感谢!!!
具体参考的博客或指南如下:
基于 Jenkins 的 CI/CD (一) 强烈推荐阳明老师的博客
kubernetes实践:安装jenkins slave
kubernetes Jenkins gitlab搭建CI/CD环境 (二)
01 [从这里开始]Jenkins CI解决方案