502 BAD Gateway | ingress nginx with cert-manager

Question

I am using nginx ingress v1.1.3 in azure AKS v1.21. My traffic is getting terminated at nginx ingress for all other domains and it is working as expected.

Recently I added cert-manager to manage certificate, but when implemented i am getting 502 for this particular ingress and same is working if i use my custom certificate.

Here's my ingress configuration:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: new-dns
  namespace: application-production
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "off"
    nginx.ingress.kubernetes.io/ssl-passthrough: "false"
    cert-manager.io/cluster-issuer: "letsencrypt"
    nginx.ingress.kubernetes.io/connection-proxy-header: upgrade
    nginx.ingress.kubernetes.io/http2-push-preload: "true"
    nginx.ingress.kubernetes.io/proxy-http-version: "1.1"
    nginx.ingress.kubernetes.io/ssl-prefer-server-ciphers: "true"
spec:
  tls:
  - hosts:
    - xxx-xxxxxx.com
    secretName: new-dns
  rules:
  - host: xxx-xxx.com
    http:
      paths:
      - pathType: Prefix
        path: /
        backend:
          service:
            name: my-backend
            port:
              number: 8080

Logs of nginx when configuration applied

I0507 02:25:05.236731       7 admission.go:149] processed ingress via admission controller {testedIngressLength:11 testedIngressTime:0.076s renderingIngressLength:11 renderingIngressTime:0.001s admissionTime:1.2MBs testedConfigurationSize:0.077}
I0507 02:25:05.236761       7 main.go:100] "successfully validated configuration, accepting" ingress="application-production/anew-dns"
I0507 02:25:05.258742       7 store.go:427] "Found valid IngressClass" ingress="application-production/anew-dns" ingressclass="nginx"
I0507 02:25:05.259236       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"application-production", Name:"anew-dns", UID:"a95d83e7-51ad-472d-a5a2-5f4cc6352e81", APIVersion:"networking.k8s.io/v1", ResourceVersion:"137003252", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0507 02:25:05.260169       7 controller.go:159] "Configuration changes detected, backend reload required"
I0507 02:25:05.418604       7 controller.go:176] "Backend successfully reloaded"
I0507 02:25:05.418745       7 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"ingress-nginx", Name:"ingress-nginx-controller-756f546d89-69xnq", UID:"bdc7b594-4070-465c-b7a9-afc846e3e7a9", APIVersion:"v1", ResourceVersion:"136713253", FieldPath:""}): type: 'Normal' reason: 'RELOAD' NGINX reload triggered due to a change in configuration
I0507 02:25:22.295245       7 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"application-production", Name:"anew-dns", UID:"a95d83e7-51ad-472d-a5a2-5f4cc6352e81", APIVersion:"networking.k8s.io/v1", ResourceVersion:"137003343", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

Error after sometime

2022/05/07 02:03:44 [error] 12307#12307: *4399339 upstream prematurely closed connection while reading response header from upstream, client: xx.xx.xx.xx, server: xxxx.xxxx.xxx, request: "GET / HTTP/2.0", upstream: "http:/xx.xx.xx.xx:8080/", host: "xxxx.xxxx.xxxx"

Answer 1

I found out that ingress and cert manager is setup correctly there was issue in my backend.

Since LetsEncrypt root cert is expired and I am calling axios which is giving invalid cert hence no response was returned to nginx ingress.

Solution: Upgrade openssl version to 1.1.0 or later.