iptables模式下的kube-proxy无法正常工作
[英]kube-proxy in iptables mode is not working
问题描述
I have 
我有
- Kubernetes: v.1.1.1 Kubernetes：v.1.1.1
- iptables v1.4.21 iptables v1.4.21
- kernel: 4.2.0-18-generic which come with Ubuntu wily 内核：Ubuntu wily随附的4.2.0-18-generic
- Networking is done via L2 VLAN terminated on switch 通过终止于交换机的L2 VLAN进行网络连接
- no cloud provider 没有云提供商
what I do 我所做的
I'm experimenting with iptables mode for kube-proxy. 我正在尝试使用iptables模式进行kube-proxy。 I have enabled it with --proxy_mode=iptables argument. 我用--proxy_mode=iptables参数启用了它。 It seems some rule is missing: 似乎缺少一些规则：
iptables -t nat -nvL
Chain PREROUTING (policy ACCEPT 8 packets, 459 bytes)
pkts bytes target prot opt in out source destination
2116 120K KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 2 packets, 120 bytes)
pkts bytes target prot opt in out source destination
718 45203 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain POSTROUTING (policy ACCEPT 5 packets, 339 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service traffic requiring SNAT */ mark match 0x4d415351
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp dpt:31195 MARK set 0x4d415351
0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp dpt:31195
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp dpt:30873 MARK set 0x4d415351
0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp dpt:30873
Chain KUBE-SEP-5IXMK7UWPGVTWOJ7 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.8 0.0.0.0/0 /* mngbox/jumpbox:ssh */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */ tcp to:10.116.160.8:22
Chain KUBE-SEP-BNPLX5HQYOZINWEQ (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */ tcp to:10.116.161.6:8086
Chain KUBE-SEP-CJMHKLXPTJLTE3OP (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.254.2 0.0.0.0/0 /* default/kubernetes: */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes: */ tcp to:10.116.254.2:6443
Chain KUBE-SEP-GSM3BZTEXEBWDXPN (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.7 0.0.0.0/0 /* kube-system/kube-dns:dns */ MARK set 0x4d415351
0 0 DNAT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */ udp to:10.116.160.7:53
Chain KUBE-SEP-OAYOAJINXRPUQDA3 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.7 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */ tcp to:10.116.160.7:53
Chain KUBE-SEP-PJJZDQNXDGWM7MU6 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.5 0.0.0.0/0 /* default/docker-registry-fe:tcp */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */ tcp to:10.116.160.5:443
Chain KUBE-SEP-RWODGLKOVWXGOHUR (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */ tcp to:10.116.161.6:8083
Chain KUBE-SEP-WE3Z7KMHA6KPJWKK (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.161.6 0.0.0.0/0 /* kube-system/monitoring-grafana: */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-grafana: */ tcp to:10.116.161.6:8080
Chain KUBE-SEP-YBQVM4LA4YMMZIWH (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.161.3 0.0.0.0/0 /* kube-system/monitoring-heapster: */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-heapster: */ tcp to:10.116.161.3:8082
Chain KUBE-SEP-YMZS7BLP4Y6MWTX5 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.9 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */ tcp to:10.116.160.9:5000
Chain KUBE-SEP-ZDOOYAKDERKR43R3 (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 10.116.160.10 0.0.0.0/0 /* default/kibana-logging: */ MARK set 0x4d415351
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kibana-logging: */ tcp to:10.116.160.10:5601
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-JRXTEHDDTAFMSEAS tcp -- * * 0.0.0.0/0 10.116.0.48 /* kube-system/monitoring-grafana: cluster IP */ tcp dpt:80
0 0 KUBE-SVC-CK6HVV5A27TDFNIA tcp -- * * 0.0.0.0/0 10.116.0.188 /* kube-system/monitoring-influxdb:api cluster IP */ tcp dpt:8086
0 0 KUBE-SVC-DKEW3YDJFV3YJLS2 tcp -- * * 0.0.0.0/0 10.116.0.6 /* infra/docker-registry-backend:docker-registry-backend cluster IP */ tcp dpt:5000
0 0 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.116.0.2 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
0 0 KUBE-SVC-WEHLQ23XZWSA5ZX3 tcp -- * * 0.0.0.0/0 10.116.0.188 /* kube-system/monitoring-influxdb:http cluster IP */ tcp dpt:8083
0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.1.142 /* default/docker-registry-fe:tcp cluster IP */ tcp dpt:443
0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 MARK set 0x4d415351
0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
0 0 KUBE-SVC-XZFGDLM7GMJHZHOY tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/docker-registry-fe:tcp external IP */ tcp dpt:443 ADDRTYPE match dst-type LOCAL
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.116.0.2 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-7IHGTXJ4CF2KVXJZ tcp -- * * 0.0.0.0/0 10.116.1.126 /* kube-system/monitoring-heapster: cluster IP */ tcp dpt:80
0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.1.175 /* mngbox/jumpbox:ssh cluster IP */ tcp dpt:2345
0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 MARK set 0x4d415351
0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
0 0 KUBE-SVC-GLKZVFIDXOFHLJLC tcp -- * * 0.0.0.0/0 10.116.254.3 /* mngbox/jumpbox:ssh external IP */ tcp dpt:2345 ADDRTYPE match dst-type LOCAL
0 0 KUBE-SVC-6N4SJQIF3IX3FORG tcp -- * * 0.0.0.0/0 10.116.0.1 /* default/kubernetes: cluster IP */ tcp dpt:443
0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.0.233 /* default/kibana-logging: cluster IP */ tcp dpt:8888
0 0 MARK tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 MARK set 0x4d415351
0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 PHYSDEV match ! --physdev-is-in ADDRTYPE match src-type !LOCAL
0 0 KUBE-SVC-B6ZEWWY2BII6JG2L tcp -- * * 0.0.0.0/0 10.116.254.3 /* default/kibana-logging: external IP */ tcp dpt:8888 ADDRTYPE match dst-type LOCAL
0 0 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-SVC-6N4SJQIF3IX3FORG (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-CJMHKLXPTJLTE3OP all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kubernetes: */
Chain KUBE-SVC-7IHGTXJ4CF2KVXJZ (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-YBQVM4LA4YMMZIWH all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-heapster: */
Chain KUBE-SVC-B6ZEWWY2BII6JG2L (3 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-ZDOOYAKDERKR43R3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/kibana-logging: */
Chain KUBE-SVC-CK6HVV5A27TDFNIA (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-BNPLX5HQYOZINWEQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:api */
Chain KUBE-SVC-DKEW3YDJFV3YJLS2 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-YMZS7BLP4Y6MWTX5 all -- * * 0.0.0.0/0 0.0.0.0/0 /* infra/docker-registry-backend:docker-registry-backend */
Chain KUBE-SVC-ERIFXISQEP7F7OF4 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-OAYOAJINXRPUQDA3 all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns-tcp */
Chain KUBE-SVC-GLKZVFIDXOFHLJLC (4 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-5IXMK7UWPGVTWOJ7 all -- * * 0.0.0.0/0 0.0.0.0/0 /* mngbox/jumpbox:ssh */
Chain KUBE-SVC-JRXTEHDDTAFMSEAS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-WE3Z7KMHA6KPJWKK all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-grafana: */
Chain KUBE-SVC-TCOU7JCQXEZGVUNU (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-GSM3BZTEXEBWDXPN all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/kube-dns:dns */
Chain KUBE-SVC-WEHLQ23XZWSA5ZX3 (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-RWODGLKOVWXGOHUR all -- * * 0.0.0.0/0 0.0.0.0/0 /* kube-system/monitoring-influxdb:http */
Chain KUBE-SVC-XZFGDLM7GMJHZHOY (4 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SEP-PJJZDQNXDGWM7MU6 all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/docker-registry-fe:tcp */
When I do request to the service ip, in my case it's 10.116.0.2 I got an error 当我向服务ip请求时，在我的情况下是10.116.0.2，我收到了一个错误
;; connection timed out; no servers could be reached
while when I do request to the 10.116.160.7 server it's working fine. 而当我向10.116.160.7服务器请求时，它工作正常。 I can see that traffic is not directed to kube-proxy rules at all, so there is something missing probably. 我可以看到流量根本不针对kube-proxy规则，因此可能缺少某些内容。
I will highly appreciate any hint about missing rule 我将非常感谢有关缺少规则的任何提示
EDIT Ive updated my initial request with missing information requested by thokin, he pointed to the really good way to debug the iptables rules for kube-proxy, and I could identify my problem with: 编辑 Ive用thokin请求的缺少信息更新了我的初始请求，他指出了调试kube-proxy iptables规则的真正好方法，并且我可以通过以下方式确定我的问题：
for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.160.7 -j LOG --log-prefix "DBG@$c: "; done
for c in PREROUTING OUTPUT POSTROUTING; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done
Then I've executed following commands: # nslookup kubernetes.default.svc.psc01.cluster 10.116.160.7 Server: 10.116.160.7 Address: 10.116.160.7#53 然后我执行了以下命令：＃nslookup kubernetes.default.svc.psc01.cluster 10.116.160.7服务器：10.116.160.7地址：10.116.160.7＃53
Name: kubernetes.default.svc.psc01.cluster
Address: 10.116.0.1
# nslookup kubernetes.default.svc.psc01.cluster 10.116.0.2
;; connection timed out; no servers could be reached
As a result I've got different "source" address and outgoing interface: 结果，我得到了不同的“源”地址和传出接口：
[701768.263847] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=12436 PROTO=UDP SPT=54501 DPT=53 LEN=62
[702620.454211] DBG@OUTPUT: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62
[702620.454224] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.176.1 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=22733 PROTO=UDP SPT=28704 DPT=53 LEN=62
[702626.318258] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
[702626.318263] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
[702626.318266] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
[702626.318270] DBG@OUTPUT: IN= OUT=bond1.300 SRC=10.116.250.252 DST=10.116.0.2 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
[702626.318284] DBG@POSTROUTING: IN= OUT=docker0 SRC=10.116.250.252 DST=10.116.160.7 LEN=82 TOS=0x00 PREC=0x00 TTL=64 ID=30608 PROTO=UDP SPT=39443 DPT=53 LEN=62
So, by adding the route 因此，通过添加路线
ip route add 10.116.0.0/23 dev docker0
Now it's working fine! 现在一切正常！
1 个解决方案
解决方案1
5 已采纳 2015-11-30 21:37:31
For future, the results of iptables-save are much easier to read (to me anyway). 对于将来， iptables-save的结果更容易阅读（无论如何对我来说）。
I don't see anything missing here. 我在这里看不到任何遗漏。
KUBE-SERVICES traps 10.116.0.2 port 53/UDP and passes it to KUBE-SVC-TCOU7JCQXEZGVUNU KUBE-SERVICES捕获10.116.0.2端口53 / UDP并将其传递给KUBE-SVC-TCOU7JCQXEZGVUNU
KUBE-SVC-TCOU7JCQXEZGVUNU has just one endpoint so jumps to KUBE-SEP-GSM3BZTEXEBWDXPN KUBE-SVC-TCOU7JCQXEZGVUNU只有一个端点，因此跳至KUBE-SEP-GSM3BZTEXEBWDXPN
KUBE-SEP-GSM3BZTEXEBWDXPN DNATs to 10.116.160.7 port 53/UDP KUBE-SEP-GSM3BZTEXEBWDXPN DNAT至10.116.160.7端口53 / UDP
If you assert that 10.116.160.7 works while 10.116.0.2 does not, that is strange indeed. 如果您断言10.116.160.7有效，而10.116.0.2无效，那确实很奇怪。 It suggests that the iptables rules are not triggering at all. 这表明iptables规则根本没有触发。 Are you testing from the node itself or from a container? 您是从节点本身还是从容器进行测试？
What networking are you using? 您正在使用什么网络？ L3 (underlay?) Flannel? L3（衬里？）绒布？ OVS? OVS？ Something else? 还有吗
What cloud provider (if any)? 什么云提供商（如果有）？
First step to debug: run: for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done 调试的第一步：运行： for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done for c in PREROUTING OUTPUT; do iptables -t nat -I $c -d 10.116.0.2 -j LOG --log-prefix "DBG@$c: "; done
That will log any packets that iptables sees to your service IP. 这会将iptables看到的所有数据包记录到您的服务IP。 Now look at dmesg . 现在看dmesg 。
声明:本站的技术帖子网页，遵循CC BY-SA 4.0协议，如果您需要转载，请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.
相关问题
iptables模式下的kube-proxy在路由中不起作用 - kube-proxy in iptables mode is not working in routing
kube-proxy 不更新 iptables - kube-proxy does not update iptables
'systemctl stop kube-proxy'将不清理iptables - ’systemctl stop kube-proxy‘ will not clean the iptables
kube-proxy无法安装iptables规则 - kube-proxy failed to install iptables rule
假设 iptables 代理问题，如何修复 kube-proxy 未知代理模式“”？ - How to fix kube-proxy Unknown proxy mode "", assuming iptables proxy issue?
在iptables模式下使用kube-proxy无法从节点外部访问Kubernetes服务 - Can't reach Kubernetes service from outside of node when kube-proxy in iptables mode
如何删除kube-proxy添加的iptables规则？ - How to delete iptables rules added by kube-proxy?
Kubernetes kube-proxy iptables 规则似乎是多余的 - Kubernetes kube-proxy iptables rules seem to be redundant
kube-proxy：缺少服务的iptables-DNAT规则 - kube-proxy: iptables-DNAT-rules for services missing
具有 IPVS 模式的 Kube-proxy 不保持连接 - Kube-proxy with IPVS mode doesn't keep a connection
相关标签