使用 nginx 入口在 k8s 中设置 rabbitmq controller

Question

I am trying to setup rabbitmq, deployed as a pod, in k8s cluster . Also did setup Nginx ingress controller in order to have external access for the services in the cluster by using URLs.

Here is the working config through which I am able to access the rabbitmq management console :

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-gateway
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /  
spec:
  tls:
  - hosts:
    - ***ABC***.com
    secretName: gateway-tls-secret
  rules:
  - host: ***ABC***.com 
    http:
      paths:
      - backend:
          serviceName: ie-rabbitmq
          servicePort: 15672
        path: /

Logs in Nginx controller

10.202.3.59 - - [24/Dec/2019:06:59:19 +0000] "GET /api/auth HTTP/2.0" 200 57 "https://***ABC***.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 324 0.003 [ie-poc-ie-rabbitmq-15672] [] 10.244.5.235:15672 57 0.003 200 eed95f6d3ee6bddae7a7128b4b500152

10.202.3.59 - - [24/Dec/2019:06:59:19 +0000] "GET /js/tmpl/login.ejs?0.16274381270760774 HTTP/2.0" 200 630 "https://***ABC***.com/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 48 0.001 [ie-poc-ie-rabbitmq-15672] [] 10.244.5.235:15672 630 0.001 200 75c43c0e3e3d8de715c4ffa540a4b0a8

But when I change the backend.path to /rabbit from / , the service is not reachable .

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-gateway
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /  
spec:
  tls:
  - hosts:
    - ***ABC***.com
    secretName: gateway-tls-secret
  rules:
  - host: ***ABC***.com 
    http:
      paths:
      - backend:
          serviceName: ie-rabbitmq
          servicePort: 15672
        path: /rabbit

From Nginx controller logs what I see is, for this config the request is being directed to [upstream-default-backend] 10.244.3.84:8080 but ideally it should re-direct the request to [ie-poc-ie-rabbitmq-15672] [] 10.244.5.235:15672 which is observed when the request was successful.

10.202.3.59 - - [24/Dec/2019:06:57:15 +0000] "GET /api/auth HTTP/2.0" 404 21 "https://i***ABC***.com/rabbit" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 57 0.001 [upstream-default-backend] [] 10.244.3.84:8080 21 0.001 404 2a1d6b6abf1b7ff03884f275c4a15c14

10.202.3.59 - - [24/Dec/2019:06:57:15 +0000] "GET /js/tmpl/login.ejs?0.4076380641124395 HTTP/2.0" 404 21 "https://***ABC***.com/rabbit" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36" 47 0.001 [upstream-default-backend] [] 10.244.3.84:8080 21 0.001 404 df534e5e9b2e5aabb9fa6bb272d4b5e9

Can someone help me to understand why the second config is not working/loading rabbitmq config ?

Answer 1

Try adding the management.path_prefix = /rabbit to your Rabbitmq configmap. That fixed the issue for me.

Answer 2

Try below the NGINX Ingress Rewrite rule, it works for me:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-gateway
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: rmq-rabbitmq-ha
              servicePort: 15672
            path: /rabbit/(.*)

And here is the screenshot:

By the way, if you don't need the subpath, the Ingress config file is this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-gateway
spec:
  rules:
    - http:
        paths:
          - backend:
              serviceName: rmq-rabbitmq-ha
              servicePort: 15672
            path: /

Answer 3

I solved this by serving RabbitMQ at the root path on a subdomain as host, configured with an Azure DNS Zone.

First I used Helm v3 to install RabbitMQ on my AKS k8s cluster:

helm install rabbitmq-resource -f .\rabbitmq.production-values.yaml bitnami/rabbitmq

Unfortunately the Healthcheck API in the chart is configured to use the root path of the provided host. As I am already working with an static IP for the Ingress controller and also with a DNS Zone to bind my custom domain(s) to the cluster it is easy to set an extra alias for the static Ingress IP and configure this as subdomain host in the Ingress resource.

So in the RabbitMQ config values which I applied in the previous command with the file rabbitmq.production-values.yaml, amke sure you use the root path at the following values:

...
rabbitmq:
  ...
  extraConfiguration: |-
    ...
    #management.path_prefix = /

...
ingress:
  ...
  enabled: true
  path: /

In the Azure Portal go to your k8s specific cluster ResourceGroup, it has the name convention:

MC_<primary_cluster_resource_group>_<cluster_name>_<region>

Go to the Public IP address that is bound to your Ingress controller. then go to Settings > Configuration and click on 'Create alias record', fill in a subdomain name of an existing DNS Zone. For example rabbitmq.yourdomain.com

Now edit your Ingress resource so that it looks somehow like the following:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-resource-handler
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /$1
spec:
  tls:
  - hosts:
    - yourdomain.com
    - rabbitmq.yourdomain.com
    secretName: tls-secret
  rules:
  - host: yourdomain.com
    http:
      paths:
      - backend:
          serviceName: website
          servicePort: 80
        path: /(.*)
  - host: rabbitmq.yourdomain.com
    http:
      paths:
      - backend:
          serviceName: rabbitmq-resource
          servicePort: 15672
        path: /(.*)

Note that via the main namespace of your domain you can serve a website or other service on the root path and via the subdomain you can serve RabbitMQ (Management), also via the root path without interfering with the Healthcheck API.

Now apply the Ingress resource and everything will be working smoothly!

Answer 4

I fixed this issue by introducing 'management.path_prefix = /rabbit-mgmt/' in RabbitMQ additionalConfig.

    "rabbitmq" = {
    "additionalPlugins" = [
      "rabbitmq_management",
      "rabbitmq_peer_discovery_k8s",
    ]
    "additionalConfig" = <<-EOT
    management.path_prefix = /rabbit-mgmt/
    EOT      
                         
  }