How to fix cert-manager responses to Let's Encrypt ACME challenges when using client certificate authentication on Kubernetes with nginx ingress?

Question

We've set-up a new ingress route that requires TLS certificate authentication, and we have placed it on its own subdomain, but we are finding that cert-manager is unable to issue a certificate for it.

Using the examples provided here, we generated the CA cert and CA key, and then configured the client certificate: https://github.com/kubernetes/ingress-nginx/tree/master/docs/examples/auth/client-certs

In the logs, I can see that all of the .acme-challenge requests are returning a 403. I am guessing that nginx is rejecting the requests because Let's Encrypt can't present a client certificate for the challenge request. What do I need to do in order to bypass the client cert requirement for ACME?

Answer 1

The issue here was that we actually had misconfigured the nginx.ingress.kubernetes.io/auth-tls-secret annotation. It must be in namespace/name format -- where namespace is the namespace that contains the secret containing the client CA certificate, and name is the name of that secret -- but we were only providing the name since the secret is in the same namespace with the ingress.

I was able to diagnose the issue by dumping the nginx ingress controller config to nginx.conf.txt with:

  kubectl exec <NAME OF INGRESS CONTROLLER POD> -n <INGRESS NAMESPACE> -- nginx -T | tee nginx.conf.txt

(Adapted from https://docs.nginx.com/nginx-ingress-controller/troubleshooting/#checking-the-generated-config ).

This included the following snippet:

    ## start server the.hostname.com
    server {
        server_name the.hostname.com ;
        
        listen 80;
        
        set $proxy_upstream_name "-";
        set $pass_access_scheme $scheme;
        set $pass_server_port $server_port;
        set $best_http_host $http_host;
        set $pass_port $pass_server_port;
        
        listen 443  ssl http2;
        
        # PEM sha: 66c07c44ba9637a23cd3d7b6ebce958e08a52ccb
        ssl_certificate                         /etc/ingress-controller/ssl/default-fake-certificate.pem;
        ssl_certificate_key                     /etc/ingress-controller/ssl/default-fake-certificate.pem;
        
        ssl_certificate_by_lua_block {
            certificate.call()
        }
        
        # Location denied, reason: invalid format (namespace/name) found in 'the-secret-name'
        return 403;
        
    }
    ## end server the.hostname.com

The key is these two lines:

        # Location denied, reason: invalid format (namespace/name) found in 'the-secret-name'
        return 403;

This pointed me to the annotation for the secret name. Once I fixed that, ACME worked properly.