Can't upgrade websocket connection in Kubernetes using Nginx-ingress

Question

I'm trying to connect to my Mosquitto broker over websockets, but I'm not able to do it because the connection doesn't upgrade. The mosquitto broker expose the port 9001 to allow websocket connections and it is running behind a Kubernetes Cluster with nginx-ingress controllers.

$ kubectl get ingress mosquitto
NAME        HOSTS   ADDRESS   PORTS   AGE
mosquitto   *                 80      14m

.

$kubectl get service

NAME                      TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)             AGE

mosquitto                 ClusterIP   10.108.206.11    <none>        9001/TCP,1883/TCP   12m

Mosquitto.yaml:

apiVersion: apps/v1beta1
kind: Deployment
metadata:
  name: mosquitto
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      imagePullSecrets:
        - name: abb-login
      containers:
        - name: mosquitto
          image: ***/mosquitto:k8s2
          imagePullPolicy: Always
          ports:
            - containerPort: 9001
              protocol: TCP
            - containerPort: 1883
              protocol: TCP
          resources: {}

---
apiVersion: v1
kind: Service
metadata:
  name: mosquitto
spec:
  ports:
    - name: "9001"
      port: 9001
      targetPort: 9001
      protocol: TCP
    - name: "1883"
      port: 1883
      targetPort: 1883
      protocol: TCP
  selector:
    app: mosquitto

Mosquitto.conf:

allow_duplicate_messages false

connection_messages true

log_dest stdout stderr
log_timestamp true
log_type all

persistence false

listener 1883

allow_anonymous true

listener 9001
protocol websockets

allow_anonymous false

auth_plugin /usr/lib/mosquitto-auth-plugin/auth-plugin.so
auth_opt_backends http
auth_opt_http_ip 127.0.0.1
auth_opt_http_getuser_uri /api/mosquitto/users
auth_opt_http_superuser_uri /api/mosquitto/admins
auth_opt_http_aclcheck_uri /api/mosquitto/permissions
auth_opt_acl_cacheseconds 1
auth_opt_auth_cacheseconds 0

Ingress.yaml:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: mosquitto
  annotations:
    nginx.org/websocket-services: mosquitto
spec:
  rules:
    - http:
        paths:
          - path: /mosquitto-ws
            backend:
              serviceName: mosquitto
              servicePort: 80

Error from the client:

MqttException (0) - java.io.IOException: WebSocket Response header: Incorrect upgrade.
opc-ua-adapter_1  |     at org.eclipse.paho.client.mqttv3.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:38)
opc-ua-adapter_1  |     at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:715)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
opc-ua-adapter_1  |     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
opc-ua-adapter_1  |     at java.base/java.lang.Thread.run(Thread.java:834)

Kubernetes ingress-nginx pod logs:

192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "GET /mosquitto-ws HTTP/1.1" 308 171 "-" "-" 218 0.000 [default-mosquitto-9001] - - - - 5db23bb19698ac94612ff6ebac265bed
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "\x88\x84\xDDi+\x5C\xECY\x1Bl" 400 157 "-" "-" 0 0.000 [] - - - - 2b8f177f0f62389ba7d918f9c36ee72e
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "GET /mosquitto-ws HTTP/1.1" 308 171 "-" "-" 218 0.000 [default-mosquitto-9001] - - - - c99fe7606530ae938297e227e34084c0
192.168.39.1 - [192.168.39.1] - - [27/Feb/2019:09:59:14 +0000] "\x88\x84dB5aUr\x05Q" 400 157 "-" "-" 0 0.000 [] - - - - 375ec1ac17cc3e0f7595cf8c1cc752c3

Answer 1

Try to increase proxy-read-timeout and proxy-send-timeout on your mosquito ingress definition.

See the NGinx Ingress doc: https://kubernetes.github.io/ingress-nginx/user-guide/miscellaneous/#websockets