Need help troubleshooting Istio IngressGateway HTTP ERROR 503

问题

My Test Environment Cluster has the following configurations :

Global Mesh Policy (Installed as part of cluster setup by our org) : output of kubectl describe MeshPolicy default

Name:         default
Namespace:
Labels:       operator.istio.io/component=Pilot
              operator.istio.io/managed=Reconcile
              operator.istio.io/version=1.5.6
              release=istio
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"authentication.istio.io/v1alpha1","kind":"MeshPolicy","metadata":{"annotations":{},"labels":{"operator.istio.io/component":...
API Version:  authentication.istio.io/v1alpha1
Kind:         MeshPolicy
Metadata:
  Creation Timestamp:  2020-07-23T17:41:55Z
  Generation:          1
  Resource Version:    1088966
  Self Link:           /apis/authentication.istio.io/v1alpha1/meshpolicies/default
  UID:                 d3a416fa-8733-4d12-9d97-b0bb4383c479
Spec:
  Peers:
    Mtls:
Events:  <none>

The above configuration I believe enables services to receive connections in mTls mode.

DestinationRule : Output of kubectl describe DestinationRule commerce-mesh-port -n istio-system

Name:         commerce-mesh-port
Namespace:    istio-system
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"commerce-mesh-port","namespace"...
API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
Metadata:
  Creation Timestamp:  2020-07-23T17:41:59Z
  Generation:          1
  Resource Version:    33879
  Self Link:           /apis/networking.istio.io/v1beta1/namespaces/istio-system/destinationrules/commerce-mesh-port
  UID:                 4ef0d49a-88d9-4b40-bb62-7879c500240a
Spec:
  Host:  *
  Ports:
    Name:      commerce-mesh-port
    Number:    16443
    Protocol:  TLS
  Traffic Policy:
    Tls:
      Mode:  ISTIO_MUTUAL
Events:      <none>

Istio Ingress-Gateway :

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: finrpt-gateway
  namespace: finrpt
spec:
  selector:
    istio: ingressgateway # use Istio's default ingress gateway
  servers:
  - port:
      name: https
      number: 443
      protocol: https
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"
  - port:
      name: http
      number: 80
      protocol: http
    tls:
      httpsRedirect: true
    hosts:
    - "*"

I created a secret to be used for TLS and using that to terminate the TLS traffic at the gateway (as configured in mode SIMPLE)

Next, I configured my VirtualService in the same namespace and did a URL match for HTTP :

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: finrpt-virtualservice
  namespace: finrpt
spec:
  hosts:
  - "*"
  gateways:
  - finrpt-gateway
  http:
  - match:
    - queryParams:
        target:
          exact: "commercialprocessor"
      ignoreUriCase: true
    route:
    - destination:
        host: finrpt-commercialprocessor
        port:
          number: 8118

The Service CommercialProcessor (ClusterIP) is expecting traffic on HTTP/8118.

With the above setting in place, when I browse to the External IP of my Ingress-Gateway, first I get a certificate error (expected as I am using self-signed for testing) and then on proceeding I get HTTP Error 503.

I am not able to find any useful logs in the gateway, I am wondering if the gateway is unable to communicate to my VirtualService in plaintext (TLS termination) and it is expecting https but I have put it as http? Any help is highly appreciated, I am very new to Istio and I think I might be missing something naive here.

My expectation is : I should be able to hit the Gateway with https, gateway does the termination and forwards the unencrypted traffic to the destination configured in the VirtualService on HTTP port based on URL regex match ONLY (I have to keep URL match part constant here).

回答1:

As 503 often occurs and it´s hard to find the issue I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.

Examples with 503 error:

Istio 503:s between (Public) Gateway and Service
IstIO egress gateway gives HTTP 503 error
Istio Ingress Gateway with TLS termination returning 503 service unavailable
how to terminate ssl at ingress-gateway in istio?
Accessing service using istio ingress gives 503 error when mTLS is enabled

Common cause of 503 errors from istio documentation:

https://istio.io/docs/ops/best-practices/traffic-management/#avoid-503-errors-while-reconfiguring-service-routes
https://istio.io/docs/ops/common-problems/network-issues/#503-errors-after-setting-destination-rule
https://istio.io/latest/docs/concepts/traffic-management/#working-with-your-applications

Few things I would check first:

Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be <protocol>[-<suffix>] as mentioned in istio documentation.
Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
Check if your namespace is injected with kubectl get namespace -L istio-injection
If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.

Hope you find this useful.

来源：https://stackoverflow.com/questions/63153829/need-help-troubleshooting-istio-ingressgateway-http-error-503

标签

http

Kubernetes

Istio

azure-aks