Kubernetes challenge waiting for http-01 propagation: dial tcp: no such host

Question

I am trying to create a kube.netes cluster namespace with auto generated DNS for ingress, secured with Let's Encrypt TLS certificates. Unfortunately i'm running in some trouble and do not know where to look for the solution.

Deployment is being done with a multi-stage yaml pipeline into an AKS cluster, i've setup an nginx ingress controller and cert-manager, both in a separate namespace. The deployment succeeds and everything seems to be running, but the exposed hostnames from the ingress are not reachable. When taking a look at the certificates i see the following

Name:         letsencrypt-tls-cd
Namespace:    myApp-dev
Labels:       app.kubernetes.io/instance=myApp
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cd
              app.kubernetes.io/version=9.3.0
              helm.sh/chart=cd-1.0.0
Annotations:  <none>
API Version:  cert-manager.io/v1alpha3
Kind:         Certificate
Metadata:
  Creation Timestamp:  2020-06-15T11:59:53Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  myApp-cd
    UID:                   a6cbbf69-749e-4dd1-81cc-37a817051690
  Resource Version:        1218430
  Self Link:               /apis/cert-manager.io/v1alpha3/namespaces/myApp-dev/certificates/letsencrypt-tls-cd
  UID:                     46ac0acb-71bf-4dbc-a376-c024e92d68ca
Spec:
  Dns Names:
    cd-myApp-dev.dev
  Issuer Ref:
    Group:      cert-manager.io
    Kind:       Issuer
    Name:       letsencrypt-prod
  Secret Name:  letsencrypt-tls-cd
Status:
  Conditions:
    Last Transition Time:  2020-06-15T11:59:53Z
    Message:               ***Waiting for CertificateRequest "letsencrypt-tls-cd-95531636" to complete***
    Reason:                InProgress
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  GeneratedKey  57m   cert-manager  Generated a new private key
  Normal  Requested     57m   cert-manager  Created new CertificateRequest resource "letsencrypt-tls-cd-95531636"

Looking into the certificate request:

Name:         letsencrypt-tls-cd-95531636
Namespace:    myApp-dev
Labels:       app.kubernetes.io/instance=myApp
              app.kubernetes.io/managed-by=Helm
              app.kubernetes.io/name=cd
              app.kubernetes.io/version=9.3.0
              helm.sh/chart=cd-1.0.0
Annotations:  cert-manager.io/certificate-name: letsencrypt-tls-cd
              cert-manager.io/private-key-secret-name: letsencrypt-tls-cd
API Version:  cert-manager.io/v1alpha3
Kind:         CertificateRequest
Metadata:
  Creation Timestamp:  2020-06-15T11:59:54Z
  Generation:          1
  Owner References:
    API Version:           cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Certificate
    Name:                  letsencrypt-tls-cd
    UID:                   46ac0acb-71bf-4dbc-a376-c024e92d68ca
  Resource Version:        1218442
  Self Link:               /apis/cert-manager.io/v1alpha3/namespaces/myApp-dev/certificaterequests/letsencrypt-tls-cd-95531636
  UID:                     2bef5e93-6722-43c0-bd2c-283d70334b1c
Spec:
  Csr:  mySecret
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
Status:
  Conditions:
    Last Transition Time:  2020-06-15T11:59:54Z
    Message:               Waiting on certificate issuance from order myApp-dev/letsencrypt-tls-cd-95531636-1679437339: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:
  Type    Reason        Age   From          Message
  ----    ------        ----  ----          -------
  Normal  OrderCreated  58m   cert-manager  Created Order resource myApp-dev/letsencrypt-tls-cd-95531636-1679437339

And the challenge:

Name:         letsencrypt-tls-cm-1259919220-2936945618-694921812
Namespace:    myApp-dev
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1alpha3
Kind:         Challenge
Metadata:
  Creation Timestamp:  2020-06-15T11:59:55Z
  Finalizers:
    finalizer.acme.cert-manager.io
  Generation:  1
  Owner References:
    API Version:           acme.cert-manager.io/v1alpha2
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  letsencrypt-tls-cm-1259919220-2936945618
    UID:                   4d8eab8e-449b-494e-a751-912a77671223
  Resource Version:        1218492
  Self Link:               /apis/acme.cert-manager.io/v1alpha3/namespaces/myApp-dev/challenges/letsencrypt-tls-cm-1259919220-2936945618-694921812
  UID:                     8b355336-309a-4192-83b7-41397ebc20ac
Spec:
  Authz URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/5253543313
  Dns Name:   cm-myApp-dev.dev
  Issuer Ref:
    Group:  cert-manager.io
    Kind:   Issuer
    Name:   letsencrypt-prod
  Key:      0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI.qZ3FGlVmwRY6MwBNqUR5iktM1fJWdXxFWZYFOpjSUkQ
  Solver:
    http01:
      Ingress:
        Class:  nginx
        Pod Template:
          Metadata:
          Spec:
            Node Selector:
              kubernetes.io/os:  linux
  Token:                         0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI
  Type:                          http-01
  URL:                           https://acme-v02.api.letsencrypt.org/acme/chall-v3/5253543313/1eUG0g
  Wildcard:                      false
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cm-myApp-dev.dev/.well-known/acme-challenge/0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI': Get "http://cm-myApp-dev.dev/.well-known/acme-challenge/0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI": dial tcp: lookup cm-myApp-dev.dev on 10.0.0.10:53: no such host
  State:       pending
Events:
  Type    Reason     Age    From          Message
  ----    ------     ----   ----          -------
  Normal  Started    2m15s  cert-manager  Challenge scheduled for processing
  Normal  Presented  2m14s  cert-manager  Presented challenge using http-01 challenge mechanism

I'm quite new to kube.netes and don't know where to look to fix the error bellow, any help is greatly appreciated.

Waiting for http-01 challenge propagation: failed to perform self check GET request 'http://cm-myApp-dev.dev/.well-known/acme-challenge/0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI': Get "http://cm-myApp-dev.dev/.well-known/acme-challenge/0USdpDsQg7_NY1FB138oj6O3AtVVKn6rkdxUSBQk4KI": dial tcp: lookup cm-myApp-dev.dev on 10.0.0.10:53: no such host

Looking in the ingress controller i get the following error:

    7 controller.go:1374] Error getting SSL certificate "myApp-dev/letsencrypt-tls-cd": local SSL certificate myApp-dev/letsencrypt-tls-cd was not found
W0616 06:24:29.033235       7 controller.go:1119] Error getting SSL certificate "myApp-dev/letsencrypt-tls-cm": local SSL certificate myApp-dev/letsencrypt-tls-cm was not found. Using default certificate
W0616 06:24:29.033264       7 controller.go:1374] Error getting SSL certificate "myApp-dev/letsencrypt-tls-cd": local SSL certificate myApp-dev/letsencrypt-tls-cd was not found
I0616 06:24:50.355937       7 status.go:275] updating Ingress myApp-dev/cm-acme-http-solver-9z88h status from [] to [{10.240.0.252 } {10.240.1.58 }]
W0616 06:24:50.363181       7 controller.go:1119] Error getting SSL certificate "myApp-dev/letsencrypt-tls-cm": local SSL certificate myApp-dev/letsencrypt-tls-cm was not found. Using default certificate
W0616 06:24:50.363346       7 controller.go:1374] Error getting SSL certificate "myApp-dev/letsencrypt-tls-cd": local SSL certificate myApp-dev/letsencrypt-tls-cd was not found
I0616 06:24:50.363514       7 event.go:278] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"myApp-dev", Name:"cm-acme-http-solver-9z88h", UID:"1b53f4dc-1b52-4f11-9cd0-6ffe1d0d9d40", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1451371", FieldPath:""}): type: 'Normal' reason: 'UPDATE' Ingress myApp-dev/cm-acme-http-solver-9z88h

Answer 1

You can refer this link to configure cert manager at AKS. It will automatically create the TLS secret too, once the certificate gets validated and will attain ready state

Answer 2

The problem was that the top level domain name we were using was not valid, therefore the ingress didn't refer to a valid domain and threw an error. Creating a valid top level domain and implementing it in our deployment solved the problem.

Answer 3

If someone Googles this, then know that this issue can be also caused by DNS caching in your Kubernetes cluster. In this case, it is a transient error, but in some contexts speed could be important (eg if you are a managed service provider).

I wrote about it here but in summary.

• cert-manager would emit the "no such host" error for a while, and eventually succeed
• my coredns ConfigMap (in kube-system namespace) stipulated local DNS resolvers, and a 30 sec cache
• you can fix the delay by (1) removing the cache , and (2) pointing the resolver to Google DNS (or another, depending on your needs)

Hope this pointer is helpful to someone.

Answer 4

Remember to add DNS records to the domain, such as A and CNAME to route traffic to the Kube.netes load balancer.

Eg cm-myApp-dev.dev or any other subdomains.