Nginx-Ingress not picking up certificate from cert-manager

Question

I'm currently trying to setup an application in K8S behind an nginx-ingress. The Certs should be generated by cert-manager and Let's Encrypt (Staging for now).

The application is in namespace prod, nginx-ingress-controller in namespace nginx and cert-manager lives in cert-manager namespace.

We setup a ClusterIssuer for Let's Encrypt staging and successfully generated a certificate (we can see it in the secrets and certificate resource). However, nginx-ingress-controller is still answering with the Kubernetes Ingress Controller Fake Certificate.

Here are some technical details:

Ingress

❯ kubectl describe ingress/forgerock
Name:             forgerock
Labels:           <none>
Namespace:        prod
Address:          someaws-id.elb.eu-central-1.amazonaws.com
Ingress Class:    <none>
Default backend:  <default>
TLS:
  sslcertciam terminates ciam.test.fancycorp.com
Rules:
  Host                Path  Backends
  ----                ----  --------
  ciam.test.fancycorp.com
                      /am/json/authenticate                                             am:80 (10.0.2.210:8081)
                      ...
                      /am/extlogin                                                      am:80 (10.0.2.210:8081)
Annotations:          cert-manager.io/cluster-issuer: letsencrypt-stage
                      haproxy.router.openshift.io/cookie_name: route
                      kubernetes.io/ingress.class: nginx
                      nginx.ingress.kubernetes.io/affinity: cookie
                      nginx.ingress.kubernetes.io/body-size: 64m
                      nginx.ingress.kubernetes.io/enable-cors: false
                      nginx.ingress.kubernetes.io/proxy-body-size: 64m
                      nginx.ingress.kubernetes.io/proxy-buffer-size: 16k
                      nginx.ingress.kubernetes.io/proxy-read-timeout: 600
                      nginx.ingress.kubernetes.io/proxy-send-timeout: 600
                      nginx.ingress.kubernetes.io/send-timeout: 600
                      nginx.ingress.kubernetes.io/session-cookie-hash: sha1
                      nginx.ingress.kubernetes.io/session-cookie-name: route
                      nginx.ingress.kubernetes.io/ssl-redirect: true
Events:               <none>

Issuer:

❯ kubectl describe clusterissuer/letsencrypt-stage
Name:         letsencrypt-stage
Namespace:
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2022-09-12T07:26:05Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:acme:
          .:
          f:email:
          f:privateKeySecretRef:
            .:
            f:name:
          f:server:
          f:solvers:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2022-09-12T07:26:05Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:acme:
          .:
          f:lastRegisteredEmail:
          f:uri:
        f:conditions:
    Manager:         controller
    Operation:       Update
    Subresource:     status
    Time:            2022-09-12T07:26:06Z
  Resource Version:  17749318
  UID:               fcbcbfff-b875-4ac4-805b-65ab0b4e1a93
Spec:
  Acme:
    Email:            admin@fancycorp.com
    Preferred Chain:
    Private Key Secret Ref:
      Name:  letsencrypt-stage
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
    Solvers:
      http01:
        Ingress:
          Class:  nginx
Status:
  Acme:
    Last Registered Email:  admin@fancycorp.com
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/68184363
  Conditions:
    Last Transition Time:  2022-09-12T07:26:06Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

Certificate:

❯ kubectl describe cert/sslcertciam
Name:         sslcertciam
Namespace:    prod
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2022-09-12T07:40:04Z
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:ownerReferences:
          .:
          k:{"uid":"2a0af8f2-8166-4a8e-bb50-fd0aa906f844"}:
      f:spec:
        .:
        f:dnsNames:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:secretName:
        f:usages:
    Manager:      controller
    Operation:    Update
    Time:         2022-09-12T07:40:04Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:notAfter:
        f:notBefore:
        f:renewalTime:
        f:revision:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2022-09-12T07:40:07Z
  Owner References:
    API Version:           networking.k8s.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  forgerock
    UID:                   2a0af8f2-8166-4a8e-bb50-fd0aa906f844
  Resource Version:        17753197
  UID:                     2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
Spec:
  Dns Names:
    ciam.test.fancycorp.com
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       ClusterIssuer
    Name:       letsencrypt-stage
  Secret Name:  sslcertciam
  Usages:
    digital signature
    key encipherment
Status:
  Conditions:
    Last Transition Time:  2022-09-12T07:40:07Z
    Message:               Certificate is up to date and has not expired
    Observed Generation:   1
    Reason:                Ready
    Status:                True
    Type:                  Ready
  Not After:               2022-12-11T06:40:05Z
  Not Before:              2022-09-12T06:40:06Z
  Renewal Time:            2022-11-11T06:40:05Z
  Revision:                1
Events:                    <none>

Secret:

❯ kubectl describe secret/sslcertciam
Name:         sslcertciam
Namespace:    prod
Labels:       <none>
Annotations:  cert-manager.io/alt-names: ciam.test.fancycorp.com
              cert-manager.io/certificate-name: sslcertciam
              cert-manager.io/common-name: ciam.test.fancycorp.com
              cert-manager.io/ip-sans:
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: ClusterIssuer
              cert-manager.io/issuer-name: letsencrypt-stage
              cert-manager.io/uri-sans:

Type:  kubernetes.io/tls

Data
====
tls.crt:  5741 bytes
tls.key:  1675 bytes

Certificate Request:

❯ kubectl describe certificaterequests/sslcertciam-p6qpg
Name:         sslcertciam-p6qpg
Namespace:    prod
Labels:       <none>
Annotations:  cert-manager.io/certificate-name: sslcertciam
              cert-manager.io/certificate-revision: 1
              cert-manager.io/private-key-secret-name: sslcertciam-ztc8q
API Version:  cert-manager.io/v1
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2022-09-12T07:40:05Z
  Generate Name:       sslcertciam-
  Generation:          1
  Managed Fields:
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:cert-manager.io/certificate-name:
          f:cert-manager.io/certificate-revision:
          f:cert-manager.io/private-key-secret-name:
        f:generateName:
        f:ownerReferences:
          .:
          k:{"uid":"2484d1fe-5b80-4cbc-b2f8-7f4276e15a37"}:
      f:spec:
        .:
        f:issuerRef:
          .:
          f:group:
          f:kind:
          f:name:
        f:request:
        f:usages:
    Manager:      controller
    Operation:    Update
    Time:         2022-09-12T07:40:05Z
    API Version:  cert-manager.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:certificate:
        f:conditions:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2022-09-12T07:40:06Z
  Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  sslcertciam
    UID:                   2484d1fe-5b80-4cbc-b2f8-7f4276e15a37
  Resource Version:        17753174
  UID:                     2289de7b-f43f-4859-816b-b4a9794846ec
Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-75947cd847-7gndz
    authentication.kubernetes.io/pod-uid:
      91415540-9113-4456-86d2-a0e28478718a
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   ClusterIssuer
    Name:   letsencrypt-stage
  Request:  xxx
  UID:      5be755b9-711c-49ac-a962-6b3a3f80d16e
  Usages:
    digital signature
    key encipherment
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Certificate:  <base64-encoded-cert>
  Conditions:
    Last Transition Time:  2022-09-12T07:40:05Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2022-09-12T07:40:06Z
    Message:               Certificate fetched from issuer successfully
    Reason:                Issued
    Status:                True
    Type:                  Ready
Events:                    <none>

Curl:

❯ curl -v https://ciam.test.fancycorp.com/am/extlogin/ -k
*   Trying xxx.xxx.xxx.xxx:443...
* Connected to ciam.test.fancycorp.com (xxx.xxx.xxx.xxx) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  start date: Sep 12 07:43:15 2022 GMT
*  expire date: Sep 12 07:43:15 2023 GMT
*  issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x126811e00)
> GET /am/extlogin/ HTTP/2
> Host: ciam.test.fancycorp.com
> user-agent: curl/7.79.1
> accept: */*
...

---

Update 1:

When running kubectl ingress-nginx certs --host ciam.test.fancycorp.com , I am also getting the Fake Certificate returned.

Answer 1

Found the issue and solution...

There was another ingress defined in another namespace that did define the same hostname, but failed to link to a proper secret with the TLS cert. When I deleted that one, it immediately worked.

Lessons learned: Be aware of impacts from other namespaces!