kubeadm kubedns错误。 无法访问外部网络或其他Pod
[英]kubeadm kubedns error. could not access external network or other pods
问题描述
When using a self hosted kubeadm in ubuntu, I could not access other pods and external network from within k8s pod but am able to access using regular docker containers. 
在ubuntu中使用自托管的kubeadm时，我无法从k8s容器内访问其他容器和外部网络，但能够使用常规docker容器进行访问。
I tried with different types of pod network including calico, weave and flannel. 我尝试了不同类型的Pod Network，包括印花布，编织和法兰绒。
I followed the debugging instructinos from here without any success, below is the logs. 我从这里开始遵循调试指令，但没有成功，下面是日志。
$ kubectl exec -ti busybox -- nslookup kubernetes.default
Server: 10.96.0.10
Address 1: 10.96.0.10
nslookup: can't resolve 'kubernetes.default'
$ kubectl exec busybox cat /etc/resolv.conf
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5
$ kubectl get pods --namespace=kube-system -l k8s-app=kube-dns
NAME READY STATUS RESTARTS AGE
kube-dns-2425271678-9zwtd 3/3 Running 0 12m
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c kubedns
I0823 16:02:58.407162 6 dns.go:48] version: 1.14.3-4-gee838f6
I0823 16:02:58.408957 6 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I0823 16:02:58.409223 6 server.go:113] FLAG: --alsologtostderr="false"
I0823 16:02:58.409248 6 server.go:113] FLAG: --config-dir="/kube-dns-config"
I0823 16:02:58.409288 6 server.go:113] FLAG: --config-map=""
I0823 16:02:58.409301 6 server.go:113] FLAG: --config-map-namespace="kube-system"
I0823 16:02:58.409309 6 server.go:113] FLAG: --config-period="10s"
I0823 16:02:58.409325 6 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I0823 16:02:58.409333 6 server.go:113] FLAG: --dns-port="10053"
I0823 16:02:58.409370 6 server.go:113] FLAG: --domain="cluster.local."
I0823 16:02:58.409387 6 server.go:113] FLAG: --federations=""
I0823 16:02:58.409401 6 server.go:113] FLAG: --healthz-port="8081"
I0823 16:02:58.409411 6 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I0823 16:02:58.409434 6 server.go:113] FLAG: --kube-master-url=""
I0823 16:02:58.409451 6 server.go:113] FLAG: --kubecfg-file=""
I0823 16:02:58.409458 6 server.go:113] FLAG: --log-backtrace-at=":0"
I0823 16:02:58.409470 6 server.go:113] FLAG: --log-dir=""
I0823 16:02:58.409478 6 server.go:113] FLAG: --log-flush-frequency="5s"
I0823 16:02:58.409489 6 server.go:113] FLAG: --logtostderr="true"
I0823 16:02:58.409496 6 server.go:113] FLAG: --nameservers=""
I0823 16:02:58.409521 6 server.go:113] FLAG: --stderrthreshold="2"
I0823 16:02:58.409533 6 server.go:113] FLAG: --v="2"
I0823 16:02:58.409544 6 server.go:113] FLAG: --version="false"
I0823 16:02:58.409559 6 server.go:113] FLAG: --vmodule=""
I0823 16:02:58.409728 6 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I0823 16:02:58.467505 6 server.go:198] Skydns metrics enabled (/metrics:10055)
I0823 16:02:58.467640 6 dns.go:147] Starting endpointsController
I0823 16:02:58.467810 6 dns.go:150] Starting serviceController
I0823 16:02:58.557166 6 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I0823 16:02:58.557335 6 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I0823 16:02:58.968454 6 dns.go:174] Waiting for services and endpoints to be initialized from apiserver...
I0823 16:02:59.468406 6 dns.go:171] Initialized services and endpoints from apiserver
I0823 16:02:59.468698 6 server.go:129] Setting up Healthz Handler (/readiness)
I0823 16:02:59.469064 6 server.go:134] Setting up cache handler (/cache)
I0823 16:02:59.469305 6 server.go:120] Status HTTP port 8081
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c dnsmasq
I0823 16:02:59.445525 11 main.go:76] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000}
I0823 16:02:59.445741 11 nanny.go:86] Starting dnsmasq [-k --cache-size=1000 --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]
I0823 16:02:59.820424 11 nanny.go:108] dnsmasq[38]: started, version 2.76 cachesize 1000
I0823 16:02:59.820546 11 nanny.go:108] dnsmasq[38]: compile time options: IPv6 GNU-getopt no-DBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset auth no-DNSSEC loop-detect inotify
I0823 16:02:59.820596 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0823 16:02:59.820623 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0823 16:02:59.820659 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0823 16:02:59.820736 11 nanny.go:108] dnsmasq[38]: reading /etc/resolv.conf
I0823 16:02:59.820762 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain ip6.arpa
I0823 16:02:59.820788 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain in-addr.arpa
I0823 16:02:59.820825 11 nanny.go:108] dnsmasq[38]: using nameserver 127.0.0.1#10053 for domain cluster.local
I0823 16:02:59.820850 11 nanny.go:108] dnsmasq[38]: using nameserver 8.8.8.8#53
I0823 16:02:59.820928 11 nanny.go:108] dnsmasq[38]: read /etc/hosts - 7 addresses
I0823 16:02:59.821193 11 nanny.go:111]
W0823 16:02:59.821212 11 nanny.go:112] Got EOF from stdout
$ kubectl logs --namespace=kube-system $(kubectl get pods --namespace=kube-system -l k8s-app=kube-dns -o name) -c sidecar
ERROR: logging before flag.Parse: I0823 16:03:00.789793 26 main.go:48] Version v1.14.3-4-gee838f6
ERROR: logging before flag.Parse: I0823 16:03:00.790052 26 server.go:45] Starting server (options {DnsMasqPort:53 DnsMasqAddr:127.0.0.1 DnsMasqPollIntervalMs:5000 Probes:[{Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1} {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}] PrometheusAddr:0.0.0.0 PrometheusPort:10054 PrometheusPath:/metrics PrometheusNamespace:kubedns})
ERROR: logging before flag.Parse: I0823 16:03:00.790121 26 dnsprobe.go:75] Starting dnsProbe {Label:kubedns Server:127.0.0.1:10053 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
ERROR: logging before flag.Parse: I0823 16:03:00.790419 26 dnsprobe.go:75] Starting dnsProbe {Label:dnsmasq Server:127.0.0.1:53 Name:kubernetes.default.svc.cluster.local. Interval:5s Type:1}
Below is the etc/resolv.conf from the master. 以下是来自主服务器的etc / resolv.conf。
$ cat /etc/resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.8.8
$ kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.3", GitCommit:"2c2fe6e8278a5db2d15a013987b53968c743f2a1", GitTreeState:"clean", BuildDate:"2017-08-03T06:43:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Below is the etc/resolv.conf from worker node where the pod is running 以下是运行Pod的工作程序节点中的etc / resolv.conf
# Dynamic resolv.conf(5) file for glibc resolver(3) generated by resolvconf(8)
# DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN
nameserver 8.8.4.4
nameserver 8.8.8.
Here is the output of sudo iptables -n -L 这是sudo iptables -n -L的输出
Chain INPUT (policy ACCEPT)
target prot opt source destination
cali-INPUT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:Cz_u1IQiXIMmKD4c */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain FORWARD (policy DROP)
target prot opt source destination
cali-FORWARD all -- 0.0.0.0/0 0.0.0.0/0 /* cali:wUHhoiAYhphO9Mso */
DOCKER-USER all -- 0.0.0.0/0 0.0.0.0/0
DOCKER-ISOLATION all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
DOCKER all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
WEAVE-NPC all -- 0.0.0.0/0 0.0.0.0/0
NFLOG all -- 0.0.0.0/0 0.0.0.0/0 state NEW nflog-group 86
DROP all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ctstate RELATED,ESTABLISHED
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
cali-OUTPUT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
KUBE-SERVICES all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
KUBE-FIREWALL all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- 0.0.0.0/0 0.0.0.0/0
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp -- 0.0.0.0/0 10.96.252.131 /* default/redis-cache-service:redis has no endpoints */ tcp dpt:6379 reject-with icmp-port-unreachable
REJECT tcp -- 0.0.0.0/0 10.96.252.131 /* default/redis-cache-service:cluster has no endpoints */ tcp dpt:16379 reject-with icmp-port-unreachable
REJECT tcp -- 0.0.0.0/0 10.105.180.126 /* default/redis-pubsub-service:redis has no endpoints */ tcp dpt:6379 reject-with icmp-port-unreachable
REJECT tcp -- 0.0.0.0/0 10.105.180.126 /* default/redis-pubsub-service:cluster has no endpoints */ tcp dpt:16379 reject-with icmp-port-unreachable
Chain WEAVE-NPC (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
ACCEPT all -- 0.0.0.0/0 224.0.0.0/4
WEAVE-NPC-DEFAULT all -- 0.0.0.0/0 0.0.0.0/0 state NEW
WEAVE-NPC-INGRESS all -- 0.0.0.0/0 0.0.0.0/0 state NEW
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 ! match-set weave-local-pods dst
Chain WEAVE-NPC-DEFAULT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-k?Z;25^M}|1s7P3|H9i;*;MhG dst
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-iuZcey(5DeXbzgRFs8Szo]+@p dst
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 match-set weave-4vtqMI+kx/2]jD%_c0S%thO%V dst
Chain WEAVE-NPC-INGRESS (1 references)
target prot opt source destination
Chain cali-FORWARD (1 references)
target prot opt source destination
cali-from-wl-dispatch all -- 0.0.0.0/0 0.0.0.0/0 /* cali:X3vB2lGcBrfkYquC */
cali-to-wl-dispatch all -- 0.0.0.0/0 0.0.0.0/0 /* cali:UtJ9FnhBnFbyQMvU */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:Tt19HcSdA5YIGSsw */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:9LzfFCvnpC5_MYXm */
MARK all -- 0.0.0.0/0 0.0.0.0/0 /* cali:7AofLLOqCM5j36rM */ MARK and 0xf1ffffff
cali-from-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 /* cali:QM1_joSl7tL76Az7 */ mark match 0x0/0x1000000
cali-to-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 /* cali:C1QSog3bk0AykjAO */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:DmFiPAmzcisqZcvo */ /* Host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000
Chain cali-INPUT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:i7okJZpS8VxaJB3n */ mark match 0x1000000/0x1000000
DROP 4 -- 0.0.0.0/0 0.0.0.0/0 /* cali:p8Wwvr6qydjU36AQ */ /* Drop IPIP packets from non-Calico hosts */ ! match-set cali4-all-hosts src
cali-wl-to-host all -- 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:QZT4Ptg57_76nGng */
MARK all -- 0.0.0.0/0 0.0.0.0/0 /* cali:V0Veitpvpl5h1xwi */ MARK and 0xf0ffffff
cali-from-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 /* cali:3R1g0cpvSoBlKzVr */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:efXx-pqD4s60WsDL */ /* Host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000
Chain cali-OUTPUT (1 references)
target prot opt source destination
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:YQSSJIsRcHjFbXaI */ mark match 0x1000000/0x1000000
RETURN all -- 0.0.0.0/0 0.0.0.0/0 /* cali:KRjBsKsBcFBYKCEw */
MARK all -- 0.0.0.0/0 0.0.0.0/0 /* cali:3VKAQBcyUUW5kS_j */ MARK and 0xf0ffffff
cali-to-host-endpoint all -- 0.0.0.0/0 0.0.0.0/0 /* cali:Z1mBCSH1XHM6qq0k */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:N0jyWt2RfBedKw3L */ /* Host endpoint policy accepted packet. */ mark match 0x1000000/0x1000000
Chain cali-failsafe-in (0 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 /* cali:wWFQM43tJU7wwnFZ */ multiport dports 22
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 /* cali:LwNV--R8MjeUYacw */ multiport dports 68
Chain cali-failsafe-out (0 references)
target prot opt source destination
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 /* cali:73bZKoyDfOpFwC2T */ multiport dports 2379
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 /* cali:QMFuWo6o-d9yOpNm */ multiport dports 2380
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 /* cali:Kup7QkrsdmfGX0uL */ multiport dports 4001
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 /* cali:xYYr5PEqDf_Pqfkv */ multiport dports 7001
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 /* cali:nbWBvu4OtudVY60Q */ multiport dports 53
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 /* cali:UxFu5cDK5En6dT3Y */ multiport dports 67
Chain cali-from-host-endpoint (2 references)
target prot opt source destination
Chain cali-from-wl-dispatch (2 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* cali:zTj6P0TIgYvgz-md */ /* Unknown interface */
Chain cali-to-host-endpoint (2 references)
target prot opt source destination
Chain cali-to-wl-dispatch (1 references)
target prot opt source destination
DROP all -- 0.0.0.0/0 0.0.0.0/0 /* cali:7KNphB1nNHw80nIO */ /* Unknown interface */
Chain cali-wl-to-host (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 /* cali:aEOMPPLgak2S0Lxs */ multiport sports 68 multiport dports 67
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 /* cali:SzR8ejPiuXtFMS8B */ multiport dports 53
cali-from-wl-dispatch all -- 0.0.0.0/0 0.0.0.0/0 /* cali:MEmlbCdco0Fefcrw */
ACCEPT all -- 0.0.0.0/0 0.0.0.0/0 /* cali:LZBoXHDOlr3ok4R3 */ /* Configured DefaultEndpointToHostAction */
1 个解决方案
解决方案1
0 2017-09-29 06:13:51
Maybe it is your iptable has old record ,because i think you use kubeadm reset or setting maybe overlay network before, Please do this thing after kubeadm reset and remove docker 也许这是您的iptable具有旧记录，因为我认为您之前使用了kubeadm重置或设置了覆盖网络，请在kubeadm重置后执行此操作并删除docker
check your ip link ip link and clean the old record 检查您的ip链接ip link并清除旧记录
ip link delete cni0 ip link delete flannel.1 (if there are other network eg weave, please delete it) ip link delete cni0 ip link delete flannel.1 （如果还有其他网络，例如编织，请删除它）
then clean the iptalbes 然后清洁iptalbes
iptables -P INPUT ACCEPT iptables -P FORWARD ACCEPT iptables -P OUTPUT ACCEPT iptables -t nat -F iptables -t mangle -F iptables -F iptables -X
then reinstall docker and kubernetes, that the pod should can get the external network 然后重新安装docker和kubernetes，该pod应该可以获取外部网络
Good Luck! 祝好运！
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
编译JDK8错误。''找不到freetype'' - Compile JDK8 error.''Could not find freetype''
如何在Ubuntu中访问局域网中其他系统的MySQL数据库连接 - How to access mysql database connection of other system in local network in ubuntu
如何解决 kubeadm init 预检错误 CRI - kubernetes - How can solve kubeadm init preflight error CRI - kubernetes
八度-严重错误。 在Linux Mint上安装 - Octave - critical error. Installation on linux mint
错误无法访问文件“ $ libdir / repmgr_funcs”没有此类文件或目录 - ERROR could not access file “$libdir/repmgr_funcs” No such file or directory
无法从本地网络上的其他计算机访问Apache虚拟主机 - Not able to access Apache virtual host from other machines on my local network
PPTP接入局域网 - PPTP access to lan network
对Docker与外部网络的连接进行故障排除 - Troubleshoot docker connectivity to external network
无法在 127.0.0.1:6379 连接到 Redis：网络无法访问 - Could not connect to Redis at 127.0.0.1:6379: Network is unreachable
如何解决 Kubeadm Inıt 错误执行阶段 certs/front-proxy-ca？ - How to solve Kubeadm Inıt error execution phase certs/front-proxy-ca?
相关标签