Error on Websocket Implementation on Azure Kubernetes Cluster Not working

问题

I am trying to make the websocket service work on Azure Kubernetes Cluster on our organization environment. My existing environment also have REST api and Angular application working on ingress with ssl. But when I added the websocket service on the ingress it is not working.

So, I tried to use Azure Free Subscription to first implement the same WITHOUT SSL. For my applications I enabled Http Routing and using the annotation addon-http-application-routing.

I am getting below error. 'ws://40.119.7.246/ws' failed: Error during WebSocket handshake: Unexpected response code: 404

Please help in validating where I am doing wrong?

Below are the details of the configuration.

Dockerfile

FROM node:alpine
WORKDIR /app
COPY package*.json /app/
RUN npm install

COPY ./ /app/
RUN npm run build

CMD ["node","./dist/server.js"]

EXPOSE 8010

socketserver.yaml - Contains Demployment & Service.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: socketserver
spec:
  replicas: 1
  selector:
    matchLabels:
      app: socketserver
  template:
    metadata:
      labels:
        app: socketserver
    spec:
      containers:
      - name: socketserver
        image: regkompella.azurecr.io/socketserver:1.0.0
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            memory: "128Mi"
            cpu: "500m"
        ports:
        - containerPort: 8010
      imagePullSecrets:
      - name: regkompella-azurecr-io
---
apiVersion: v1
kind: Service
metadata:
  name: socketserver-svc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 8010
  selector:
    app: socketserver
  type: ClusterIP
---

ingress.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    kubernetes.io/ingress.class: addon-http-application-routing
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 10m
    nginx.ingress.kubernetes.io/websocket-services: socketserver-svc
    nginx.org/websocket-services: socketserver-svc
spec:
  rules:
  - host: demosocket.com
  - http:
      paths:
      - path: /
        backend:
          serviceName: angular-application-svc
          servicePort: 80
      - path: /ws
        backend:
          serviceName: socketserver-svc
          servicePort: 80

回答1:

After reading through a lot of articles and referring some of the github forums (Added referenced articles below). I come to a point where my websocket implementation started working after doing the two things. I am not sure yet if, this is the right way to do it or not. I achieved to this solution purely on trail and error method. Hence, I request everyone who have good grasp, kindly suggest if there is a better way to solve my problem. Always take my steps with a pinch of salt.

Installed the NGINX Ingress controller from the link.

As I am using Azure Kubernetes Services, I applied the below yaml from the document.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml

Next I made necessary changes to my demo-ingress configuration for my services.

I come to know that kubernetes.io/ingress.class: addon-http-application-routing annotation doesn't support websocketing. So, had to disable it.

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: demo-ingress
  annotations:
    # this one annotation is making the websocket work.
    nginx.ingress.kubernetes.io/websocket-services: socketserver-svc

    # this one I left as-is. And not playing any role for this websocket 
    # implementation to work
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"

    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/proxy-body-size: 10m

    # I thought sticky session is also required for websocket to work. But seems 
    # this has no effect after I installed nginx ingress controller.
    # so disabled all the below annotations also.

    #nginx.org/websocket-services: socketserver-svc
    #nginx.ingress.kubernetes.io/affinity: cookie
    #nginx.ingress.kubernetes.io/affinity-mode: balanced
    #nginx.ingress.kubernetes.io/session-cookie-samesite: Strict
    #kubernetes.io/ingress.class: nginx
    #kubernetes.io/ingress.class: addon-http-application-routing

spec:
  rules:
  - host: demosocket.com
  - http:
      paths:
      - path: /ws
        backend:
          serviceName: socketserver-svc
          servicePort: 80

I tried to access through the public IP address. And I can be able to successfully send and receive messages.

ws://52.188.38.118/ws

Now, what if I want to make the websocket implementation work without installing NGINX Ingress Controller ( indicated on step 1) and want to try to use default ingress controller coming with AKS/minikube. The answer is below.

From the steps above,

a) Avoid Step 1: Installing NGINX Ingress Controller.

b) Only change that need to be made on ingress is below. Use the below annotations instead of the annotations indicated on Step 2 on the ingress yaml file. Things will start working.

# this annotation is making my web application also work if I plan to configure something in future.
nginx.ingress.kubernetes.io/ingress.class: nginx

# this one annotation is making the websocket work.
nginx.ingress.kubernetes.io/websocket-services: socketserver-svc

# by default ssl is true - as I am trying locally and want to disable ssl-# redirect. So set this to false.
nginx.ingress.kubernetes.io/ssl-redirect: "false"

# Below are just additional annotation to allow CORS etc.
nginx.ingress.kubernetes.io/cors-allow-methods: "GET, PUT, POST, DELETE, OPTIONS"
nginx.ingress.kubernetes.io/proxy-body-size: 10m

Referenced Articles:

https://medium.com/flant-com/comparing-ingress-controllers-for-kubernetes-9b397483b46b
https://kubernetes.github.io/ingress-nginx/deploy/#azure
Mr. dstrebel's comments -> https://github.com/Azure/AKS/issues/768

I typically recommend just setting up a Ingress Controller on the cluster and not enabling "http-application-routing", as there's a lot of limitations to it. The goal with HTTP Application ROuting was for users to get setup quickly with Ingress, but not really for production deployments due to the limitations of the configuration.
DenisBiondic commented on Oct 2, 2018 -> https://github.com/Azure/AKS/issues/672

I am not 100% certain, since I don't use application routing feature, but >I think it does not use the https://github.com/nginxinc/kubernetes-ingress/tree/master/examples/websocket controller but rather the https://github.com/kubernetes/ingress-nginx. In case of the latter, I think enabling session affinity with cookies might be enough: https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md#session-affinity

In your case you are using the wrong annotation which does not work with application routing ingress controller under the hood.

I welcome suggestions and best practices.

来源：https://stackoverflow.com/questions/61336969/error-on-websocket-implementation-on-azure-kubernetes-cluster-not-working

标签

websocket

kubernetes-ingress

nginx-ingress

azure-kubernetes