Kubernetes Ingress TLS not being created with headless service

Question

What I'm trying to achieve

I'm trying to deploy an elixir (phoenix) application in a microk8s cluster namespace with TLS using let's encrypt. The cluster is hosted on an AWS EC2 instance.

The problem I'm facing

• The ingress is created in the namespace
• ingress routes to the correct domain
• the application is working and displayed on the given domain

The TLS secret is not being created in the namespace and a 'default' one is created

The secrets after deploying both phoenix app and httpbin app:

me@me:~/Documents/kubernetes-test$ kubectl get secret -n production
NAME                           TYPE                                  DATA   AGE
default-token-jmgrg            kubernetes.io/service-account-token   3      20m
httpbin-tls                    kubernetes.io/tls                     2      81s

The domain is insecure, ie the TLS is not working.

Logs from the ingress controller after applying the yml files:

W0106 17:26:36.967036       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:46.445248       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
W0106 17:26:49.779680       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate
I0106 17:26:56.431925       6 status.go:281] "updating Ingress status" namespace="production" ingress="phoenix-app-ingress" currentValue=[] newValue=[{IP:127.0.0.1 Hostname: Ports:[]}]
I0106 17:26:56.443405       6 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"production", Name:"phoenix-app-ingress", UID:"REDACTED", APIVersion:"networking.k8s.io/v1beta1", ResourceVersion:"1145907", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
W0106 17:26:56.443655       6 backend_ssl.go:46] Error obtaining X.509 certificate: no object matching key "production/phoenix-app-tls" in local store
W0106 17:26:56.443781       6 controller.go:1192] Error getting SSL certificate "production/phoenix-app-tls": local SSL certificate production/phoenix-app-tls was not found. Using default certificate

The description of the created ingress, note that here at the bottom it says Successfully created Certificate "phoenix-app-tls" but the secret does not exist :

me@me:~/Documents/kubernetes-test$ kubectl describe ing phoenix-app-ingress -n production
Name:             phoenix-app-ingress
Labels:           app=phoenix-app
Namespace:        production
Address:          127.0.0.1
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  phoenix-app-tls terminates phoenix.sub.mydomain.com
Rules:
  Host                           Path  Backends
  ----                           ----  --------
  phoenix.sub.mydomain.com  
                                 /   phoenix-app-service-headless:8000 (REDACTED_IP:4000,REDACTED_IP:4000)
Annotations:                     cert-manager.io/cluster-issuer: letsencrypt
                                 nginx.ingress.kubernetes.io/cors-allow-credentials: true
                                 nginx.ingress.kubernetes.io/cors-allow-methods: GET, POST, OPTIONS
                                 nginx.ingress.kubernetes.io/cors-allow-origin: *
                                 nginx.ingress.kubernetes.io/enable-cors: true
Events:
  Type    Reason             Age                  From                      Message
  ----    ------             ----                 ----                      -------
  Normal  CreateCertificate  29m                  cert-manager              Successfully created Certificate "phoenix-app-tls"
  Normal  Sync               8m43s (x3 over 29m)  nginx-ingress-controller  Scheduled for sync

Resources

The deployment yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: phoenix-app
  labels:
    app: phoenix-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: phoenix-app
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: phoenix-app
    spec:
      containers:
      - name: phoenix-app
        image: REDACTED
        imagePullPolicy: Always
        command: ["./bin/hello", "start"]
        lifecycle:
          preStop:
            exec:
              command: ["./bin/hello", "stop"]
        ports:
        - containerPort: 4000
        env:
        - name: POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        envFrom:
        - configMapRef:
            name: phoenix-app-config
        - secretRef:
            name: phoenix-app-secrets
      imagePullSecrets:
      - name: gitlab-pull-secret

The service yml:

apiVersion: v1
kind: Service
metadata:
  name: phoenix-app-service-headless
  labels:
    app: phoenix-app
spec:
  clusterIP: None
  selector:
    app: phoenix-app
  ports:
  - name: http
    port: 8000
    targetPort: 4000 # The exposed port by the phoenix app

Note: I removed my actual domain

The ingress yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: phoenix-app-ingress
  labels:
    app: phoenix-app
  annotations:
    nginx.ingress.kubernetes.io/enable-cors: "true"
    nginx.ingress.kubernetes.io/cors-allow-methods: "GET, POST, OPTIONS"
    nginx.ingress.kubernetes.io/cors-allow-origin: "*"
    nginx.ingress.kubernetes.io/cors-allow-credentials: "true"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - "phoenix.sub.mydomain.com"
    secretName: phoenix-app-tls
  rules:
  - host: "phoenix.sub.mydomain.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: phoenix-app-service-headless
            port:
              number: 8000 # Same port as in service.yml

Tested with different service

I deployed a sample service using httpbin (is not a headless service) and the TLS works fine in the same namespace. Here are the resources that I used to deploy it:

deplyoment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  replicas: 1
  selector: 
    matchLabels:
      app: httpbin
      version: v1
  template:
    metadata:
      labels:
        app: httpbin
        version: v1
    spec:
      containers:
      - image: docker.io/kennethreitz/httpbin
        imagePullPolicy: Always
        name: httpbin
        ports:
        - containerPort: 80

The service yml:

apiVersion: v1
kind: Service
metadata:
  name: httpbin
  labels:
    app: httpbin
spec:
  ports:
  - name: http
    port: 8000
    targetPort: 80
  selector:
    app: httpbin

The ingress yml:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: httpbin
  labels:
    app: httpbin
  annotations:
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - "httpbin.sub.mydomain.com"
    secretName: httpbin-tls
  rules:
  - host: "httpbin.sub.mydomain.com" # This is a subdomain we want to route these requests to
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: httpbin
            port:
              number: 8000

My best guess is that it has something to do with the fact that the service is headless, but I have no clue as to how I can resolve the issue.

Answer 1

I found out that you can actually check for certificates with kubectl: kubectl get certificate -n production

The status of this certificate was READY = FALSE.

I checked the description: kubectl describe certificate <certificate_name> -n production

At the bottom it said: Too many certificates have been created in the last 164 hours for this exact domain.

I just changed the domain and voila. It works.