我的 pod 無法連接到私有子網

Question

[Interface:vetha13c9067] 05:57:57.851421 IP 10.28.0.7 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:cbr0] 05:57:57.851421 IP 10.28.0.7 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64
[Interface:eth0] 05:57:57.851614 IP 172.18.0.25 > 172.28.10.17: ICMP echo request, id 56, seq 1, length 64

以上日志沒有任何問題10.28.0.7可以與 172.28.10.17 成功通信

[Interface:veth916b4093] 05:57:09.699334 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:cbr0] 05:57:09.699334 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64
[Interface:eth0] 05:57:09.699380 IP 10.20.4.194 > 172.28.10.17: ICMP echo request, id 28, seq 1, length 64

並且上面的日志有一些問題，因為您可以看到它的 eth0 接口請求仍然來自10.20.4.194而不是172.18.0.0/16其子網范圍。

為什么我的 pod 中的請求不是來自子網范圍 (172.18.0.0/16)？

以下是我當前的配置：

bash-5.0# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0@if135: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc noqueue state UP group default
    link/ether 76:f9:ea:bf:d7:1d brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 10.20.4.194/24 scope global eth0
       valid_lft forever preferred_lft forever
bash-5.0# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.20.4.1       0.0.0.0         UG        0 0          0 eth0
10.20.4.0       0.0.0.0         255.255.255.0   U         0 0          0 eth0
bash-5.0# ip neigh sh
10.20.4.1 dev eth0 lladdr 92:7f:08:52:f9:d4 STALE
bash-5.0# cat /etc/resolv.conf
nameserver 10.85.0.10
search default.svc.cluster.local svc.cluster.local cluster.local c.buzzdata.internal google.internal
options ndots:5
bash-5.0# nslookup kubernetes
Server:     10.85.0.10
Address:    10.85.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.85.0.1

╰─ k describe po netshoot-container
Name:               netshoot-container
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               gke-search-cluster-pool-765be39a-gkt4/172.18.0.17
Start Time:         Mon, 16 Mar 2020 14:05:32 +0900
Labels:             run=netshoot-container
Annotations:        kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container netshoot-container
Status:             Running
IP:                 10.20.4.194
Containers:
  netshoot-container:
    Container ID:  docker://0df9d4b262f926d7d89a42f58de672284a5cb5637ab951b752e0c8b34ded676a
    Image:         nicolaka/netshoot
    Image ID:      docker-pullable://nicolaka/netshoot@sha256:99d15e34efe1e3c791b0898e05be676084638811b1403fae59120da4109368d4
    Port:          <none>
    Host Port:     <none>
    Args:
      /bin/bash
    State:          Running
      Started:      Mon, 16 Mar 2020 14:05:36 +0900
    Ready:          True
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-xxxx (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  default-token-xxxx:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-xxxx
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  kubernetes.io/hostname=gke-xxxx-cluster-pool-xxxx-gkt4
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

Chain INPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-EXTERNAL-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes externally-visible service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain FORWARD (policy DROP)
target     prot opt source               destination         
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere            
ACCEPT     udp  --  anywhere             anywhere            
ACCEPT     icmp --  anywhere             anywhere            
ACCEPT     sctp --  anywhere             anywhere            

Chain OUTPUT (policy DROP)
target     prot opt source               destination         
KUBE-SERVICES  all  --  anywhere             anywhere             ctstate NEW /* kubernetes service portals */
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-EXTERNAL-SERVICES (1 references)
target     prot opt source               destination         

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  10.20.0.0/14         anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.20.0.0/14         /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-SERVICES (3 references)
target     prot opt source               destination         

Answer 1

我找到了一個原因，那是因為啟用了 IP-MASQ。

如果您遇到了 pod 無法與 172.xxx 等私有子網通信的相同問題，請通過iptables -t nat -L檢查您節點的 iptable 規則

如果你能看到 IP-MASQ 規則並且如果有RETURN all -- anywhere 172.16.0.0/12 /* ip-masq: RFC 1918 reserved range is not subject to MASQUERADE */那么下面！

https://cloud.google.com/kubernetes-engine/docs/how-to/ip-masquerade-agent

手動修改 iptable 可能是解決方案之一，但我認為最好遵循文檔。