How to create custom istio ingress gateway controller?

Question

Our GKE cluster is shared to multiple teams in company. Each team can have different public domain (and hence want to have different CA cert setup and also different ingress

Answer 1

Okay, I found the answer after looking at the code of Istio installation via helm. So, basically the istio have an official way (but not really documented in their readme.md file) to add additional gateway (ingress and egress gateway). I know that because I found this yaml file in their github repo and read the comment (also looking at the gateway chart template code for the spec and its logic).

So, I solved this by, for example, defining this values-custom-gateway.yaml file:

# Gateways Configuration
# By default (if enabled) a pair of Ingress and Egress Gateways will be created for the mesh.
# You can add more gateways in addition to the defaults but make sure those are uniquely named
# and that NodePorts are not conflicting.
# Disable specifc gateway by setting the `enabled` to false.
#
gateways:
  enabled: true

  agung-ingressgateway:
    namespace: agung-ns
    enabled: true
    labels:
      app: agung-istio-ingressgateway
      istio: agung-ingressgateway
    replicaCount: 1
    autoscaleMin: 1
    autoscaleMax: 2
    resources: {}
      # limits:
      #  cpu: 100m
      #  memory: 128Mi
      #requests:
      #  cpu: 1800m
      #  memory: 256Mi

    loadBalancerIP: ""
    serviceAnnotations: {}
    type: LoadBalancer #change to NodePort, ClusterIP or LoadBalancer if need be

    ports:
      ## You can add custom gateway ports
    - port: 80
      targetPort: 80
      name: http2
      # nodePort: 31380
    - port: 443
      name: https
      # nodePort: 31390
    - port: 31400
      name: tcp
    secretVolumes:
    - name: ingressgateway-certs
      secretName: istio-ingressgateway-certs
      mountPath: /etc/istio/ingressgateway-certs
    - name: ingressgateway-ca-certs
      secretName: istio-ingressgateway-ca-certs
      mountPath: /etc/istio/ingressgateway-ca-certs

If you take a look at yaml file above, I specified the namespace other than istio-system ns. In this case, we can have a way to customize the TLS and ca cert being used by our custom gateway. Also the agung-ingressgateway as the holder of the custom gateway controller spec is used as the gateway controller's name.

Then, i just install the istio via helm upgrade --install so that helm can intelligently upgrade the istio with additional gateway.

helm upgrade my-istio-release-name <istio-chart-folder> --install

Once it upgrades successfully, I can specify custom selector to my Gateway:

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: agung-gateway
  namespace: agung-ns
spec:
  selector:
    app: agung-istio-ingressgateway # use custom gateway
    # istio: ingressgateway # use Istio default gateway implementation
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"

Answer 2

I tried this and worked:

---
# Source: istio/charts/gateways/templates/serviceaccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: beta-ingressgateway-service-account
  namespace: beta
  labels:
    app: ingressgateway-beta
---

---
# Source: istio/charts/gateways/templates/clusterrole.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  labels:
    app: gateways
  name: ingressgateway-beta
rules:
- apiGroups: ["extensions"]
  resources: ["thirdpartyresources", "virtualservices", "destinationrules", "gateways"]
  verbs: ["get", "watch", "list", "update"]
---

---
# Source: istio/charts/gateways/templates/clusterrolebindings.yaml

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ingressgateway-beta
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ingressgateway-beta
subjects:
  - kind: ServiceAccount
    name: beta-ingressgateway-service-account
    namespace: beta
---

---
# Source: istio/charts/gateways/templates/service.yaml

apiVersion: v1
kind: Service
metadata:
  name: ingressgateway-beta
  namespace: beta
  annotations:
  labels:
    istio: ingressgateway-beta
spec:
  type: LoadBalancer
  selector:
    istio: ingressgateway-beta
  ports:
    -
      name: http
      port: 80
      targetPort: 80
    -
      name: https
      port: 443
      targetPort: 443
---

---
# Source: istio/charts/gateways/templates/deployment.yaml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ingressgateway-beta
  namespace: beta
  labels:
    istio: ingressgateway-beta
spec:
  replicas: 1
  template:
    metadata:
      labels:
        istio: ingressgateway-beta
      annotations:
        sidecar.istio.io/inject: "false"
        scheduler.alpha.kubernetes.io/critical-pod: ""
    spec:
      serviceAccountName: beta-ingressgateway-service-account
      tolerations:
      - key: "env"
        operator: "Equal"
        value: "beta"
        effect: "NoSchedule"
      nodeSelector:
        env: beta
      containers:
        - name: istio-proxy
          image: "ISTIO_PROXY_IMAGE"
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 80
            - containerPort: 443
          args:
          - proxy
          - router
          - -v
          - "2"
          - --discoveryRefreshDelay
          - '1s' #discoveryRefreshDelay
          - --drainDuration
          - '45s' #drainDuration
          - --parentShutdownDuration
          - '1m0s' #parentShutdownDuration
          - --connectTimeout
          - '10s' #connectTimeout
          - --serviceCluster
          - ingressgateway-beta
          - --zipkinAddress
          - zipkin.istio-system:9411
          - --proxyAdminPort
          - "15000"
          - --controlPlaneAuthPolicy
          - NONE
          - --discoveryAddress
          - istio-pilot.istio-system:8080
          resources:
            requests:
              cpu: 10m
          env:
          - name: POD_NAME
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.name
          - name: POD_NAMESPACE
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: metadata.namespace
          - name: INSTANCE_IP
            valueFrom:
              fieldRef:
                apiVersion: v1
                fieldPath: status.podIP
          - name: ISTIO_META_POD_NAME
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          volumeMounts:
          - name: istio-certs
            mountPath: /etc/certs
            readOnly: true
          - name: ingressgateway-beta-certs
            mountPath: "/etc/istio/ingressgateway-beta-certs"
            readOnly: true
          - name: ingressgateway-beta-ca-certs
            mountPath: "/etc/istio/ingressgateway-beta-ca-certs"
            readOnly: true
      volumes:
      - name: istio-certs
        secret:
          secretName: istio.beta-ingressgateway-service-account
          optional: true
      - name: ingressgateway-beta-certs
        secret:
          secretName: "istio-ingressgateway-beta-certs"
          optional: true
      - name: ingressgateway-beta-ca-certs
        secret:
          secretName: "istio-ingressgateway-beta-ca-certs"
          optional: true
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - amd64
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - ppc64le
          - weight: 2
            preference:
              matchExpressions:
              - key: beta.kubernetes.io/arch
                operator: In
                values:
                - s390x
---

---
# Source: istio/charts/gateways/templates/autoscale.yaml
# Source: istio/charts/gateways/templates/autoscale.yaml

apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
    name: ingressgateway-beta
    namespace: beta
spec:
    maxReplicas: 5
    minReplicas: 1
    scaleTargetRef:
      apiVersion: apps/v1beta1
      kind: Deployment
      name: ingressgateway-beta
    metrics:
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: 80
---

remember to replace ISTIO_PROXY_IMAGE, nodeSelector and tolerations

Answer 3

In fact, it is very simple. Istio's ingress is just a regular Kubernetes Service of "Load Balancer" type. So, if you want to create additional IngresGateway, you can just apply your service (you can put any ports you want):

apiVersion: v1
kind: Service
metadata:
  name: istio-ingressgateway-custom
  namespace: istio-system
  annotations:
  labels:
    chart: gateways-1.0.5
    release: istio
    heritage: Tiller
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  type: LoadBalancer
  selector:
    app: istio-ingressgateway
    istio: ingressgateway
  ports:
    -
      name: http2
      nodePort: 31381
      port: 80
      targetPort: 80
    -
      name: https
      nodePort: 31391
      port: 443
      targetPort: 555
    -
      name: tcp
      nodePort: 31401
      port: 31400
    -
      name: tcp-pilot-grpc-tls
      port: 15011
      targetPort: 15011
    -
      name: tcp-citadel-grpc-tls
      port: 8060
      targetPort: 8060
    -
      name: tcp-dns-tls
      port: 853
      targetPort: 853
    -
      name: http2-prometheus
      port: 15030
      targetPort: 15030
    -
      name: http2-grafana
      port: 15031
      targetPort: 15031
---

Considering you have this in file named "customingress.yaml", then you apply this using command:

kubectl apply -f customingress.yaml

Answer 4

This is what I use in Istio 1.4.

To generate the new istio-ingressgateway Deployment, Service and ServiceAccount in your own namespace (bookinfo in this example)

helm template install/kubernetes/helm/istio/ \
  --namespace bookinfo \
  --set global.istioNamespace=istio-system \
  -x charts/gateways/templates/deployment.yaml \
  -x charts/gateways/templates/service.yaml  \
  -x charts/gateways/templates/serviceaccount.yaml \
  --set gateways.istio-ingressgateway.enabled=true \
  --set gateways.istio-egressgateway.enabled=false \
  --set gateways.istio-ingressgateway.labels.app=custom-istio-ingressgateway \
  --set gateways.istio-ingressgateway.labels.istio=custom-ingressgateway \
  > customingress.yaml

then, apply the generated file:

kubectl apply -f customingress.yaml

Now, you can refer to this from your Gateway resource like:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
spec:
  selector:
    istio: custom-ingressgateway # use the CUSTOM istio controller
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - "*"

You can set custom Service annotations by adding it to the helm template command like this:

 --set gateways.istio-ingressgateway.serviceAnnotations.'service\.kubernetes\.io/ibm-load-balancer-cloud-provider-ip-type'=private \

Answer 5

My guess, I have not try it yet:

1. Create multiple istio-ingressgateway deployment with differnet labels, like: istio: ingressgateway1, istio: ingressgateway2, ..., and different tls keys.

2. Create multiple Gateways to use the different istio-ingressgateways.

3. Create multiple Virtualservice to use the different Gateways.