[英]CoreDNS Not resolving service url outside namespace with K8S / Minikube
I have a local cluster with minikube 1.6.2 running.我有一个运行 minikube 1.6.2 的本地集群。
All my pods are OK, I checked the logs individually, but I have 2 db, influx and postgres, are not accesible anymore from any url outside namespace.我所有的 pod 都正常,我单独检查了日志,但我有 2 db、influx 和 postgres,不能再从命名空间外的任何 url 访问。
I logged into both pods, and I can confirm that each db is OK, has data, and I can connect manually with my user / pass.我登录了两个 pod,我可以确认每个 db 都正常,有数据,我可以手动连接我的用户/密码。
Let's take the case of influx.让我们以涌入的情况为例。
kubectl exec -it -n influx blockchain-influxdb-local-fb745b98c-vbghp -- influx -username='myuser' -password="mypass" -database="mydb" -precision=rfc3339 -execute "show measurements"
gives me 4 measurements, so no pb.给了我 4 个测量值,所以没有 pb。
but when I try to connect influx from the same namespace with his local dns, I get a timeout.但是当我尝试将来自同一命名空间的涌入与他的本地 dns 连接时,我得到了超时。
➜ ~ kubectl get svc -n influx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
blockchain-influxdb-local ClusterIP 10.96.175.62 <none> 8086/TCP 19m
➜ ~ kubectl get deployments -n influx
NAME READY UP-TO-DATE AVAILABLE AGE
blockchain-influxdb-local 1/1 1 1 20m
➜ ~ kubectl get po -n influx
NAME READY STATUS RESTARTS AGE
blockchain-influxdb-local-fb745b98c-vbghp 1/1 Running 0 21m
measures-api-local-8667bb496f-4wp8d 1/1 Running 0 21m
Case where it works:工作情况:
From a pod inside the same namespace:从同一命名空间内的 pod:
curl --verbose -G 'http://blockchain-influxdb-local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
From a pod in another namespace (same namespace), with pod IP来自另一个命名空间(相同命名空间)中的 pod,具有 pod IP
curl --verbose -G '172.17.0.5:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
From a pod in another namespace (same namespace), with service IP来自另一个命名空间(相同命名空间)中的 pod,具有服务 IP
curl --verbose -G '10.96.175.62:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
But when I use local dns from outside namespace, it won't work, I get a timeout from CURL:但是当我从外部命名空间使用本地 dns 时,它不起作用,我从 CURL 得到超时:
curl --verbose -G 'blockchain-influxdb-local.influx.svc.cluster.local:8086/query?db=mydb&pretty=true' --data-urlencode 'u=myuser' --data-urlencode 'p=mypass' --data-urlencode 'precision=rfc3339' --data-urlencode 'q=show measurements'
I followed those debug step to ensure DNS is working, and had no problem, everything works.我按照这些调试步骤来确保 DNS 正常工作,没有问题,一切正常。
https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/ https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/
Inside same pod, when I ping this url, I get:在同一个 pod 中,当我 ping 这个 url 时,我得到:
root@metadata-api-local-8b4b7846b-zllb8:/go/src/gitlab.com/company/metadata_api# ping blockchain-influxdb-local.influx.svc.cluster.local
PING nc-ass-vip.sdv.fr (212.95.74.75) 56(84) bytes of data.
--- nc-ass-vip.sdv.fr ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 47ms
I don't know why is it making a reference to nc-ass-vip.sdv.fr
我不知道为什么要引用
nc-ass-vip.sdv.fr
I also tried to remove local cluster and redeploy it, also tried to update minikube to latest version (1.8.2), nothing worked.我还尝试删除本地集群并重新部署它,还尝试将 minikube 更新到最新版本(1.8.2),但没有任何效果。
I don't know what else to do...我不知道还能做什么...
Has anyone an idea ?有人有什么想法吗? I was working well for monthes, don't really know what happened.
我工作了好几个月,真的不知道发生了什么。 :(
:(
In response to @Arghya Sadhu, I post the file /etc/resolv.conf from the Influx pod:为了回应@Arghya Sadhu,我从 Influx pod 中发布了文件 /etc/resolv.conf:
nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:5
kubectl edit cm coredns -n kube-system
# Please edit the object below. Lines beginning with a '#' will be ignored,
# and an empty file will abort the edit. If an error occurs while saving this file will be
# reopened with the relevant failures.
#
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health {
lameduck 5s
}
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
ttl 30
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2020-03-19T10:59:28Z"
name: coredns
namespace: kube-system
resourceVersion: "176"
selfLink: /api/v1/namespaces/kube-system/configmaps/coredns
uid: 0797c1a9-e9db-4b4c-bc8d-4c7ecca24968
EDIT:编辑:
kubectl exec -ti dnsutils -- nslookup blockchain-influxdb-local.influx.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr canonical name = nc-ass-vip.sdv.fr.
Name: nc-ass-vip.sdv.fr
Address: 212.95.74.75
After digging into a few possibilities we came across the output for the following commands:在深入研究了几种可能性之后,我们发现了以下命令的输出:
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
$ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Output for these commandshere (adding to the end of this answer for future reference in case of link doesn't work).这些命令的输出在这里(添加到此答案的末尾以供将来参考,以防链接不起作用)。
Reviewing this output we can see that no matter what numericable.fr
is always giving positive answer to dns queries.查看此输出我们可以看到,无论
numericable.fr
总是对 dns 查询给出肯定的回答。
To avoid this situation you can change ndots entry to 1 or even 0 in your pods.为避免这种情况,您可以将 pod 中的 ndots 条目更改为 1 甚至 0。
nameserver 10.96.0.10
search influx.svc.cluster.local svc.cluster.local cluster.local numericable.fr
options ndots:0
From man pages we have:从手册页我们有:
ndots:n Sets a threshold for the number of dots which must appear in a name given to res_query(3) (see resolver(3)) before an initial absolute query will be made.
ndots:n 设置在进行初始绝对查询之前必须出现在指定给 res_query(3)(请参阅 resolver(3))的名称中的点数的阈值。 The default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name before any search list elements are appended to it.
n 的默认值是 1,这意味着如果名称中有任何点,则在将任何搜索列表元素附加到名称之前,将首先尝试将该名称作为绝对名称。 The value for this option is silently capped to 15.
此选项的值以静默方式限制为 15。
A more effective and long term solution is to add this entry in the pod/statefulset/deployment manifest as in this example:更有效和长期的解决方案是在 pod/statefulset/deployment 清单中添加此条目,如下例所示:
apiVersion: v1
kind: Pod
metadata:
namespace: default
name: dns-example
spec:
containers:
- name: test
image: nginx
dnsConfig:
options:
- name: ndots
value: "0"
Output from commands referenced for future reference:参考命令的输出以供将来参考:
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.default.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 10
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.default.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local
internet address = 10.96.72.6
ttl = 10
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Name: blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6
pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.default.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.svc.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.cluster.local, type = A, class = IN
ANSWERS:
AUTHORITY RECORDS:
-> cluster.local
origin = ns.dns.cluster.local
mail addr = hostmaster.cluster.local
serial = 1584628757
refresh = 7200
retry = 1800
expire = 86400
minimum = 30
ttl = 30
ADDITIONAL RECORDS:
------------
** server can't find blockchain-influxdb-local.influx.svc.cluster.local.cluster.local: NXDOMAIN
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr
canonical name = nc-ass-vip.sdv.fr.
ttl = 30
-> nc-ass-vip.sdv.fr
internet address = 212.95.74.75
ttl = 30
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Non-authoritative answer:
blockchain-influxdb-local.influx.svc.cluster.local.numericable.fr canonical name = nc-ass-vip.sdv.fr.
Name: nc-ass-vip.sdv.fr
Address: 212.95.74.75
pod "dnsutils" deleted
pod default/dnsutils terminated (Error)
➜ ~ kubectl run dnsutils -it --rm=true --restart=Never --image=tutum/dnsutils -- nslookup -debug blockchain-influxdb-local.influx.svc.cluster.local.
Server: 10.96.0.10
Address: 10.96.0.10#53
------------
QUESTIONS:
blockchain-influxdb-local.influx.svc.cluster.local, type = A, class = IN
ANSWERS:
-> blockchain-influxdb-local.influx.svc.cluster.local
internet address = 10.96.72.6
ttl = 30
AUTHORITY RECORDS:
ADDITIONAL RECORDS:
------------
Name: blockchain-influxdb-local.influx.svc.cluster.local
Address: 10.96.72.6
pod "dnsutils" deleted
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.