Ingress为弥补NodePort不足而生
NodePort一些不足:
• 一个端口只能一个服务使用,端口需提前规划
• 只支持4层负载均衡
nginx 动态感知pod ip 的变化,根据变化动态设置nginx 的upstream,并实现负载均衡
ingress controller 动态刷新 pod ip 列表 更新到 nginx 的配置文件
Pod与Ingress的关系
通过Service相关联
通过Ingress Controller实现Pod的负载均衡 - 支持TCP/UDP 4层和HTTP 7层
Ingress Controller
1. 部署Ingress Controller
Nginx:官方维护的Ingress Controller
部署文档:https://github.com/kubernetes/ingress-nginx/blob/master/docs/deploy/index.md
注意事项:
• 镜像地址修改成国内的:registry.aliyuncs.com/google_containers/nginx-ingress-controller:0.26.1
• 使用宿主机网络:hostNetwork: true
wget https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/mandatory.yaml
mandatory.yaml 里一共有3个configmap 一个是配置七层的,二个是配置四层的分别对应udp tcp
从apiserver 动态获取 endpoint ,从apiserver访问就必须授权,就需要创建serviceaccount,仅仅获取ip 列表,所以授予的权限 查看即可。
由于国外的镜像往往不能被 很好d 拉取。
更换配置中的默认镜像地址registry.cn-hangzhou.aliyuncs.com/benjamin-learn//nginx-ingress-controller:0.20.0
hostNetwork 字段和 container 一个层级
hostNetwork: true
创建ingress-controller 发现pod 起不来
报错提示1:
Error generating self-signed certificate: could not create temp pem file /etc/ingress-co
ntroller/ssl/default-fake-certificate.pem: open /etc/ingress-controller/ssl/default-fake-certificate.pem797363033: permission denied
解决方法:
原因:随着版本提高,安全限制越来越高,对于权限的管理也越来越精细
node节点分别 chmod -R 777 /var/lib/docker 授权任意用户有docker临时文件的任意权限
完整 ingress-controller的 yaml
mandatory.yaml
apiVersion: v1 kind: Namespace metadata: name: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: nginx-configuration namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: tcp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- kind: ConfigMap apiVersion: v1 metadata: name: udp-services namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses verbs: - get - list - watch - apiGroups: - "extensions" - "networking.k8s.io" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: apps/v1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: replicas: 1 selector: matchLabels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx template: metadata: labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx annotations: prometheus.io/port: "10254" prometheus.io/scrape: "true" spec: hostNetwork: true # wait up to five minutes for the drain of connections terminationGracePeriodSeconds: 300 serviceAccountName: nginx-ingress-serviceaccount containers: - name: nginx-ingress-controller image: registry.aliyuncs.com/google_containers/nginx-ingress-controller:0.26.1 args: - /nginx-ingress-controller - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --publish-service=$(POD_NAMESPACE)/ingress-nginx - --annotations-prefix=nginx.ingress.kubernetes.io securityContext: allowPrivilegeEscalation: true capabilities: drop: - ALL add: - NET_BIND_SERVICE # www-data -> 33 runAsUser: 33 env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 10 lifecycle: preStop: exec: command: - /wait-shutdown ---
创建ingress-service.yaml
由于nodeport 默认的端口是 30000-32767 所以 80 和443 端口想通过nodeport暴露需要修改kube-apiserver的配置,并重启kube-apiserver服务
apiVersion: v1 kind: Service metadata: name: ingress-nginx namespace: ingress-nginx labels: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx spec: type: NodePort ports: - name: http port: 80 targetPort: 80 protocol: TCP nodePort: 80 # http请求对外映射80端口 - name: https port: 443 targetPort: 443 protocol: TCP nodePort: 443 # https请求对外映射443端口 selector: app.kubernetes.io/name: ingress-nginx app.kubernetes.io/part-of: ingress-nginx ---
ingress-nginx 名称空间下的资源 皆为 ready
查看pod 所在节点 443 和 80 端口是否正常监听
符合预期 80 443 端口正常监听
2. 创建Ingress规则
创建ingress规则实际指的就是配置 nginx 的server 模块配置,通过yaml 定义的规则自动刷新到pod中的nginx 的配置中
创建一个私有仓库的secret 用于拉取 我的阿里云私有镜像
kubectl create secret docker-registry myregistry --docker-server=registry.cn-hangzhou.aliyuncs.com --docker-username=benjamin7788 --docker-password=a7260488
web-deployment.yaml 创建一个3副本的java 镜像,使用我的私有阿里云镜像
增加imagePullSecrets和 containers 一个层级
imagePullSecrets:
- name: myregistry
apiVersion: apps/v1 kind: Deployment metadata: labels: app: web name: web spec: replicas: 3 selector: matchLabels: app: web strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: labels: app: web spec: imagePullSecrets: - name: myregistry containers: - image: registry.cn-hangzhou.aliyuncs.com/benjamin-learn/java-demo:lst imagePullPolicy: Always name: java resources: {} restartPolicy: Always
web-service.yaml 将这个deployment 用nodeport 方式暴露
apiVersion: v1 kind: Service metadata: labels: app: web name: web namespace: default spec: ports: - port: 80 protocol: TCP targetPort: 8080 selector: app: web type: NodePort
ingress.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress spec: rules: - host: example.ctnrs.com http: paths: - path: / backend: serviceName: web servicePort: 80
#生产环境
这个example.ctnrs.com 域名一般是由域名运营商负责解析 其中A记录对应的值为运行ingress-controller的 node的公网ip
#测试环境
本地绑定host文件,模拟访问。
windows集群 win+r 输入drivers 进入 etc目录 修改hosts文件 增加一条记录
192.168.31.65 example.ctnrs.com
查看域名对应规则
当ingress规则创建查看ingress-controller 容器日志 发现 成功创建了一个ingress规则
进入ingress-controller 容器发现 nginx.conf 自动增加了一个server 模块的配置
成功访问java应用
请求路径:
http 请求--->node:80 --->(upstream)pod ip--->容器:8080
细节:
我们都知道nginx更新配置都需要reload 方能生效,而nginx-conroller内的nginx 采用 lua脚本将配置更新在内存中,使得不需要reload 也可以动态更新配置。
管理员 -->ingress yaml -->master(apiserver)--->service---master(apiserver)<--- ingress controller(pod ip) --->nginx.conf(lua脚本)--->upstream(pod ip) --->容器
3.根据urlL路由到多个服务
www.ctnrs.com/a 请求到a组服务器
www.ctnrs.com/b 请求到b组服务器
ingress-url.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress annotations: nginx.ingress.kubernetes.io/rewrite-target: / spec: rules: - host: example.ctnrs.com http: paths: - path: /a backend: serviceName: web servicePort: 80 - host: example.ctnrs.com http: paths: - path: /b backend: serviceName: web2 servicePort: 80
创建 web2应用
kubectl create deploy web2 --image=nginx kubectl expose deploy web2 --port=80 --target-port=80
4.基于名称的虚拟主机
ingress-vhost.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress spec: rules: - host: example1.ctnrs.com http: paths: - path: / backend: serviceName: web servicePort: 80 - host: example2.ctnrs.com http: paths: - path: / backend: serviceName: web2 servicePort: 80
绑定本地host文件 192.168.31.65 example1.ctnrs.com example1.ctnrs.com
访问example1.ctnrs.com
访问 example2.ctnrs.com
4.ingress 配置https
ingress-https.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress annotations: nginx.ingress.kubernetes.io/force-ssl-redirect: 'true' spec: tls: - hosts: - example.ctnrs.com secretName: secret-tls rules: - host: example.ctnrs.com http: paths: - path: / backend: serviceName: web servicePort: 80
访问 example.ctnrs.com 直接跳转到 https://example.ctnrs.com
选择信任后,默认使用的证书是 ingress-controller 内默认的证书
生成自签证书
cert.sh
cat > ca-config.json <<EOF { "signing": { "default": { "expiry": "87600h" }, "profiles": { "kubernetes": { "expiry": "87600h", "usages": [ "signing", "key encipherment", "server auth", "client auth" ] } } } } EOF cat > ca-csr.json <<EOF { "CN": "kubernetes", "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "Beijing", "ST": "Beijing" } ] } EOF cfssl gencert -initca ca-csr.json | cfssljson -bare ca - cat > example.ctnrs.com-csr.json <<EOF { "CN": "example.ctnrs.com", "hosts": [], "key": { "algo": "rsa", "size": 2048 }, "names": [ { "C": "CN", "L": "BeiJing", "ST": "BeiJing" } ] } EOF cfssl gencert -ca=ca.pem -ca-key=ca-key.pem -config=ca-config.json -profile=kubernetes example.ctnrs.com-csr.json | cfssljson -bare example.ctnrs.com #kubectl create secret tls example-ctnrs-com --cert=example.ctnrs.com.pem --key=example.ctnrs.com-key.pem
执行生成证书脚本 ,生成域名证书 example.ctnrs.com-key.pem example.ctnrs.com.pem
创建tls类型secret
kubectl create secret tls secret-tls --cert=/root/learn/ssl/example.ctnrs.com.pem --key=/root/learn/ssl/example.ctnrs.com -key.pemsecret/secret-tls created
再次查看证书,已经使用了 自签域名证书。
5.Annotations对Ingress个性化配置
ingress-controller的默认代理超时设置
ingress-annotations.yaml
apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: example-ingress annotations: kubernetes.io/ingress.class: "nginx" nginx.ingress.kubernetes.io/proxy-connect-timeout: "600" nginx.ingress.kubernetes.io/proxy-send-timeout: "600" nginx.ingress.kubernetes.io/proxy-read-timeout: "600" nginx.ingress.kubernetes.io/proxy-body-size: "10m" spec: rules: - host: example.ctnrs.com http: paths: - path: / backend: serviceName: web servicePort: 80
通过注解的方式更改 ingress-controller的默认代理超时时间
Ingress Controller 高可用方案
方案一:通过keepalived 设置vip访问 的 主备高可用方案,使用daemonset 控制器在每个节点部署一个 ingress controller
缺点:由于只有一台ingress controller 提供服务,所以当访问量变大,性能会产生瓶颈。
通过loadbalancer 反向代理多个 ingress-controller ,实现对多个ingress controller 的负载均衡,用户请求 lb 的ip 然后转发到 后端 ingress controller
nodeselector + daemonset +lb 完成部署
优点:这种很好的弥补了第一种方案的 不足,多台ingress controller 同时提供服务,可提供更大的 访问量。
缺点:lb 这时又会产生单点,需要对 lb 做高可用,这样比较消耗 资源。
其他主流控制器:
Traefik: HTTP反向代理、负载均衡工具
Istio:服务治理,控制入口流量
来源:https://www.cnblogs.com/benjamin77/p/12446758.html