Kubernetes 多个 nginx 入口重定向到错误的服务

Question

I want to deploy two version of my app on the same cluster. To do that I used namespace to separates them and each app have it's own ingress redirecting to it's own service. I use controller in my ingress.

To sum the architecture looks like this:

• cluster
  • namespace1
    • app1
    • service1
    • ingress1
  • namespace
    • app2
    • service2
    • ingress2

My problem is that when i'm using the external ip of the nginx-controller of the ingress2 it hits my app1

I'm using helm to deploy my app

Ingress.yaml

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: "{{ .Release.Name }}-ingress"
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
    - hosts:
      - {{ .Values.host }}
      secretName: {{ .Release.Namespace }}-cert-secret
  rules:
  - http:
    - path: /api($|/)(.*)
      backend:
        serviceName: "{{ .Release.Name }}-api-service"
        servicePort: {{ .Values.api.service.port.api }}

service.yaml

apiVersion: v1
kind: Service
metadata:
  name: "{{ .Release.Name }}-api-service"
spec:
  selector:
    app: "{{ .Release.Name }}-api-deployment"
  ports:
    - port: {{ .Values.api.service.port.api }}
      targetPort: {{ .Values.api.deployment.port.api }}
      name: 'api'

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: "{{ .Release.Name }}-api-deployment"
spec:
  replicas: 1
  selector:
    matchLabels:
      app: "{{ .Release.Name }}-api-deployment"
  template:
    metadata:
      labels:
        app: "{{ .Release.Name }}-api-deployment"
    spec:
      containers:
      - name: "{{ .Release.Name }}-api-deployment-container"
        imagePullPolicy: "{{ .Values.api.image.pullPolicy }}"
        image: "{{ .Values.api.image.repository }}:{{ .Values.api.image.tag }}"
        command: ["/bin/sh"]
        args:
        - "-c"
        - "node /app/server/app.js"
        env:
        - name: API_PORT
          value: {{ .Values.api.deployment.port.api | quote }}

values.yaml

api:
  image:
    repository: xxx
    tag: xxx
    pullPoliciy: Always
  deployment:
    port:
      api: 8080
    ressources:
      requests:
        memory: "1024Mi"
        cpu: "1000m"
  service:
    port:
      api: 80
    type: LoadBalancer

To deploy my app i run:

• helm install -n namespace1 release1 .
• helm install -n namespace2 release2 .

kubectl -n namespace1 get svc

NAME                                       TYPE           CLUSTER-IP      EXTERNAL-IP                                                               PORT(S)                                       AGE
nginx-ingress-1581005515-controller        LoadBalancer   10.100.20.183   a661e982f48fb11ea9e440eacdf86-1089217384.eu-west-3.elb.amazonaws.com   80:32256/TCP,443:32480/TCP                    37m
nginx-ingress-1581005515-default-backend   ClusterIP      10.100.199.97   <none>                                                                    80/TCP                                        37m
release1-api-service                       LoadBalancer   10.100.87.210   af6944a7b48fb11eaa3100ae77b6d-585994672.eu-west-3.elb.amazonaws.com    80:31436/TCP,8545:32715/TCP,30300:30643/TCP   33m

kubectl -n namespace2 get svc

NAME                                       TYPE           CLUSTER-IP       EXTERNAL-IP                                                               PORT(S)                                       AGE
nginx-ingress-1580982483-controller        LoadBalancer   10.100.177.215   ac7d0091648c511ea9e440eacdf86-762232273.eu-west-3.elb.amazonaws.com    80:32617/TCP,443:30459/TCP                    7h6m
nginx-ingress-1580982483-default-backend   ClusterIP      10.100.53.245    <none>                                                                    80/TCP                                        7h6m
release2-api-service                       LoadBalancer   10.100.108.190   a4605dedc490111ea9e440eacdf86-2005327771.eu-west-3.elb.amazonaws.com   80:32680/TCP,8545:32126/TCP,30300:30135/TCP   36s

When I hit the nginx-controller of the namespace2 it should hit app2 deployed in the release2 but instead it hits app1. When I hit the nginx-controller of the namespace1, as expected it hit app1.

Just for infos the order is important, it's always the first deployed app that is always hit

Why does the second load balancer isn't redirecting to my second application ?

Answer 1

The problem is that I was using the same "nginx" class for both ingress. Both nginx controller was serving the same class "nginx".

Here is the wiki of how to use mutilple nginx ingress controller: https://kubernetes.github.io/ingress-nginx/user-guide/multiple-ingress/

I end up defining my ingress class like this: kubernetes.io/ingress.class: nginx-{{ .Release.Namespace }}

And deploying my nginx controller like this: install -n $namespace nginx-$namespace stable/nginx-ingress --set "controller.ingressClass=nginx-${namespace}"

If you're not using helm to deploy your nginx-controller what you need to modify is the nginx ingress class