Websocket connection in Kubernetes cluster with nginx-ingress

Question

I'm trying to get a simple websocket connection working on my server running in a Kube.netes cluster.

Websocket connections are able to establish on my local test machine but I can't connect my client side to the server after I deploy to GKE with nginx-ingress.

ingress yaml file:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.org/websocket-services : "socket-cluster-ip-service"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/websocket-services : "socket-cluster-ip-service"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
spec:
  rules:
    - host: my-socket.com
      http:
        paths:
          - path: /
            backend:
              serviceName: socket-cluster-ip-service
              servicePort: 3000
    - host: www.my-socket.com
      http:
        paths:
          - path: /
            backend:
              serviceName: socket-cluster-ip-service
              servicePort: 3000

service yaml file:

apiVersion: v1
kind: Service
metadata:
  name: socket-cluster-ip-service
spec:
  type: ClusterIP
  selector:
    component: socket
  ports:
  - port: 3000
    targetPort: 3000

deployment yaml file:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: socket-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      component: socket
  template:
    metadata:
      labels:
        component: socket
    spec:
      containers:
      - name: socket
        image: gcr.io/my-socket/socket
        ports:
        - containerPort: 3000

express server code:

...

var server = http.createServer(app);
const WebSocket = require('ws');

const wss = new WebSocket.Server({ server: server });

wss.on('connection', function connection(ws) {
  console.log("new connection", ws)
  ws.on('message', function incoming(message) {
    console.log('received: %s', message);
  });

  ws.send('something');
});

...

client side code:

var ws = new WebSocket('ws://www.my-socket.com/');

ws.onopen = () => {
  // connection opened
  ws.send('something'); // send a message
};

ws.onmessage = (e) => {
  // a message was received
  console.log(e.data);
};

ws.onerror = (e) => {
  // an error occurred
  console.log(e.message);
};

ws.onclose = (e) => {
  // connection closed
  console.log(e.code, e.reason);
};

I keep getting the error message "Invalid Sec-WebSocket-Accept response" from the client side. But I think it's just the fact that the client could not establish websocket connection with the server.

The server side library I used is ws

I've added the following annotations to my ingress yaml file as suggested by other post and official ingress-nginx documentation but it's still not working:

nginx.org/websocket-services : "socket-cluster-ip-service"
nginx.ingress.kubernetes.io/websocket-services : "socket-cluster-ip-service"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"

Answer 1

For me it works like a charm!
I'm using kong-docker-kubernetes-ingress-controller.bintray.io/kong-ingress-controller:0.9.0 on my private kubernetes cluster.
Thus, your issue might be related to GCE.

Please also check your TLS configuration.
In your post you were mentioning ws://... rather than wss:// , which indicates you don't even use TLS. This might be rejected by your browser (eg Chrome).

Answer 2

im not on GKE however this might help the above issue: https://medium.com/k8scaleio/running-websocket-app-on-kube.netes-2e13eabb4c4f

it's to do with persistent state. I'm having the same issue works perfectly, when i run the docker image locally(docker run...) but the image deployed inside the cluster (k3d with loadbalacncer exposing its port )...web-socket will not connect. other services exposed (rest->mysql etc all work fine only the nodejs app with web-socket fails)

solution is to add annotations to your ingress controller rules:

annotations section in the yaml file:

certmanager.k8s.io/cluster-issuer: [your issuer]
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/proxy-read-timeout: "1800"
nginx.ingress.kubernetes.io/proxy-send-timeout: "1800"
nginx.ingress.kubernetes.io/rewrite-target: /
nginx.ingress.kubernetes.io/secure-backends: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/websocket-services: ws-service
nginx.org/websocket-services: ws-service