kube.netes 中的多个 ingress-nginx 未验证 webhook 无法正常工作

Question

As stated in the title, I currently have a configuration with 2 ingress-nginx v1.0.0 on gke v1.20.10.

When I deploy one alone the configuration is working and I have no issue, but when I deploy the second one the validatingwebhook and then try to deploy an ingress the 2 validatingwebhook try to evaluate the newly created ingress.

This result in this error:

**Error from server (InternalError): error when creating "ingress-example.yaml": Internal error occurred: failed calling webhook "validate.nginx-public.ingress.kubernetes.io": Post "https://ingress-nginx-controller-admission-public.ingress-nginx.svc:443/networking/v1/ingresses?timeout=10s": x509: certificate is valid for ingress-nginx-controller-admission-private, ingress-nginx-controller-admission-private.ingress-nginx.svc, not ingress-nginx-controller-admission-public.ingress-nginx.svc**

I checked and everything seems to be correctly separated, my validatingwebhook is deployed like that, the {{ ingress_type }} is a placeholder for -public or -private:

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: admission-webhook
  name: ingress-nginx-admission{{ ingress_type }}
webhooks:
  - name: validate.nginx{{ ingress_type }}.ingress.kubernetes.io
    matchPolicy: Equivalent
    objectSelector:
      matchLabels:
        ingress-nginx : nginx{{ ingress_type }}
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission{{ ingress_type }}
        path: /networking/v1/ingresses
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: controller
  name: ingress-nginx-controller-admission{{ ingress_type }}
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}

I can't seem to find a solution, there is an old github issue on that with no answer, maybe I'm doing something wrong but I just can't see it.

As asked in comment, here is the ingress-example I'm trying to deploy, this works perfectly fine with only one ingress, not with two:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: example-ingress
  annotations:
    kubernetes.io/ingress.class: nginx-private
#    external-dns.alpha.kubernetes.io/target: "IP"
  labels:
    ingress-nginx : nginx-public
spec:
  rules:
    - host: hello.MYDOMAINHERE
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: web
                port:
                  number: 8080

Answer 1

So for those that may encounter this error.

I tried different things before finding what was wrong. You have to rename all the labels but the version of the ingress-nginx, I did not think that it would break for so little, but it does. In the end I'm using something like this:

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: admission-webhook{{ ingress_type }}
  name: ingress-nginx-admission{{ ingress_type }}
webhooks:
  - name: validate.nginx{{ ingress_type }}.ingress.kubernetes.io
    matchPolicy: Equivalent
    objectSelector:
      matchLabels:
        ingress-nginx : nginx{{ ingress_type }}
    rules:
      - apiGroups:
          - networking.k8s.io
        apiVersions:
          - v1
        operations:
          - CREATE
          - UPDATE
        resources:
          - ingresses
    failurePolicy: Fail
    sideEffects: None
    admissionReviewVersions:
      - v1
    clientConfig:
      service:
        namespace: ingress-nginx
        name: ingress-nginx-controller-admission{{ ingress_type }}
        path: /networking/v1/ingresses
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/instance: ingress-nginx{{ ingress_type }}
    app.kubernetes.io/version: 1.0.0
    app.kubernetes.io/component: controller{{ ingress_type }}
  name: ingress-nginx-controller-admission{{ ingress_type }}
spec:
  type: ClusterIP
  ports:
    - name: https-webhook
      port: 443
      targetPort: webhook
      appProtocol: https
  selector:
    app.kubernetes.io/name: ingress-nginx{{ ingress_type }}

I think in this case it's really important to do the same on all the resources.

Answer 2

did this solution work for having Webhook validating base on the ingressClass that is specified in the ingrass.yaml object? I guess not. In my case, lets say I have: NamespaceA, IngressControllerA with ingressClassA and ValidatingWebHookA and in a different namespace NamespaceB, IngressControllerB with ingressClassB and ValidatingWebHookB

Now, if I create ingress with ingressClassA. ValidatingWebHookB seems to validating it and admitting it to IngressControllerA. Which is fine but if I remove ingressControllerB, then It wont work anymore. Complaining serviceB for the validatingWebhookB is not available. I was hoping they would be complelety separate from each other.