CoreDNS Not resolving service url outside namespace with K8S / Minikube

Question

I have a local cluster with minikube 1.6.2 running.

All my pods are OK, I checked the logs individually, but I have 2 db, influx and postgres, are not accesible anymore from any url outside namespace.

I logged into both pods, and I can confirm that each db is OK, has data, and I can connect manually with my user / pass.

Let's take the case of influx.

kubectl exec -it -n influx blockchain-influxdb-local-fb745b98c-vbghp -- influx -username='myuser' -password="mypass" -database="mydb" -precision=rfc3339 -execute "show measurements"

gives me 4 measurements, so no pb.

but when I try to connect influx from the same namespace with his local dns, I get a timeout.

➜ ~ kubectl get svc -n influx 
NAME                        TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)     AGE
blockchain-influxdb-local   ClusterIP   10.96.175.62   <none>        8086/TCP    19m


➜ ~ kubectl get deployments -n influx
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
blockchain-influxdb-local   1/1     1            1           20m


➜  ~ kubectl get po -n influx
NAME                                        READY   STATUS    RESTARTS   AGE
blockchain-influxdb-local-fb745b98c-vbghp   1/1     Running   0          21m
measures-api-local-8667bb496f-4wp8d         1/1     Running   0          21m

Case where it works:

From a pod inside the same namespace:

curl --verbose -G 'http://blockchain-influxdb-local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

From a pod in another namespace (same namespace), with pod IP

curl --verbose -G '172.17.0.5:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

From a pod in another namespace (same namespace), with service IP

curl --verbose -G '10.96.175.62:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

But when I use local dns from outside namespace, it won't work, I get a timeout from CURL:

 curl --verbose -G 'blockchain-influxdb-local.influx.svc.cluster.local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'

I followed those debug step to ensure DNS is working, and had no problem, everything works.

https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/

Inside same pod, when I ping this url, I get:

root@metadata-api-local-8b4b7846b-zllb8:/go/src/gitlab.com/company/metadata_api# ping blockchain-influxdb-local.influx.svc.cluster.local

PING nc-ass-vip.sdv.fr (212.95.74.75) 56(84) bytes of data.
--- nc-ass-vip.sdv.fr ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 47ms

I don't know why is it making a reference to nc-ass-vip.sdv.fr

I also tried to remove local cluster and redeploy it, also tried to update minikube to latest version (1.8.2), nothing worked.

I don't know what else to do...

Has anyone an idea ? I was working well for monthes, don't really know what happened. :(

In response to @Arghya Sadhu, I post the file /etc/resolv.conf from the Influx pod:

nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:5

kubectl edit cm coredns -n kube-system

# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2020-03-19T10:59:28Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "176"
  selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
  uid: 0797c1a9-e9db-4b4c-bc8d-4c7ecca24968

EDIT:

kubectl exec -ti dnsutils -- nslookup blockchain-influxdb-local.influx.svc.cluster.local                                         
Server:     10.96.0.10
Address:    10.96.0.10#53

Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75

Answer 1

After digging into a few possibilities we came across the output for the following commands:

$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx

$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local

$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.

Output for these commandshere (adding to the end of this answer for future reference in case of link doesn't work).

Reviewing this output we can see that no matter what numericable.fr is always giving positive answer to dns queries.

To avoid this situation you can change ndots entry to 1 or even 0 in your pods.

nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:0

From man pages we have:

ndots:n Sets a threshold for the number of dots which must appear in a name given to res_query(3) (see resolver(3)) before an initial absolute query will be made. The default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name before any search list elements are appended to it. The value for this option is silently capped to 15.

A more effective and long term solution is to add this entry in the pod/statefulset/deployment manifest as in this example:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsConfig:
    options:
      - name: ndots
        value: "0"

Output from commands referenced for future reference:

➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 10
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 10
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local 

Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.cluster.local, type = A, class = IN
    ANSWERS:
    AUTHORITY RECORDS:
    ->  cluster.local
    origin = ns.dns.cluster.local
    mail addr = hostmaster.cluster.local
    serial = 1584628757
    refresh = 7200
    retry = 1800
    expire = 86400
    minimum = 30
    ttl = 30
    ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.cluster.local: NXDOMAIN
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr
    canonical name = nc-ass-vip.sdv.fr.
    ttl = 30
    ->  nc-ass-vip.sdv.fr
    internet address = 212.95.74.75
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr   canonical name = nc-ass-vip.sdv.fr.
Name:   nc-ass-vip.sdv.fr
Address: 212.95.74.75

pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜  ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Server:     10.96.0.10
Address:    10.96.0.10#53

------------
    QUESTIONS:
    blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
    ANSWERS:
    ->  blockchain-influxdb-local.influx.svc.cluster.local
    internet address = 10.96.72.6
    ttl = 30
    AUTHORITY RECORDS:
    ADDITIONAL RECORDS:
------------
Name:   blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6

pod "dnsutils" deleted