Kubernetes pods cannot access HTTPS sites

Question

Just installed Kubernetes cluster using kubespray . 3 master nodes and 3 worker nodes + 2 haproxy nodes in front of master nodes with keepalived .

Everything works perfectly except for one thing. When I try to update packages on alpine or ubuntu inside pods:

Ign:1 https://security.debian.org/debian-security buster/updates InRelease          
Ign:2 https://deb.debian.org/debian buster InRelease                                
Err:4 https://security.debian.org/debian-security buster/updates Release                            
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. The name in the certificate does not match the expected.

Quick debugging revealed that I'm getting self-signed Traefik Default certificate from proxy in K8s

echo | openssl s_client -showcerts -servername gnupg.org -connect gnupg.org:443 2>/dev/null | openssl x509 -inform pem -noout -text

Output:

       Issuer: CN = TRAEFIK DEFAULT CERT
        Validity
            Not Before: Jun 15 14:08:39 2020 GMT
            Not After : Jun 15 14:08:39 2021 GMT
        Subject: CN = TRAEFIK DEFAULT CERT

Is there a way to disable SSL termination?

My installation is a completely fresh install of kubespray from master branch

Kubernetes version: v1.18.5

Answer 1

Found an issue in /etc/resolv.conf . search list had a domain that was pointing to a server with traefik on it. That domain's DNS zone had *.domain.com record in it. So that was the problem.