通过 GRE 隧道的 MTU 问题

Question

我面临 GRE 隧道和 MTU 的问题。 这是我的拓扑： https://user-images.githubusercontent.com/40271438/67222183-171d1b80-f42d-11e9-8ffa-f032a84b6280.png

首先，我将解释我的拓扑。

我有 3 个站点，每个站点有 2 个路由器（实际上我们有 9 个站点，但在这种情况下，只需要 3 个站点即可了解我的问题）。

• 橙色路由器是每个站点的主路由器（服务器的默认网关），灰色的是备用路由器（通过 VRRP 协议）
• 图中，每条绿色链路都是挂载了StrongSwan的IPSEC VPN隧道。
• 在这些隧道中，有一个 GRE 接口，它在 VPN IPSEC 每一侧的节点之间安装 GRE 隧道（隧道和 GRE IP 在图中为黑色）。
• 每个节点都有 Loopback 地址（图中蓝色的 IP）。
• 当然，每个节点都有LAN ip（图中绿色IP）。
• 每个节点还具有 BGP 配置，以宣布可路由网络。
• 每个路由器都在 Linux Ubuntu 18.04.3

当 2 台服务器需要通过此基础架构进行通信时，我对 MTU 有奇怪的问题。 当我从 172.16.97.2（图中的黑色服务器）ping 服务器 10.55.0.69 时，一切正常，我能够看到进出路由器的流量。

ping 10.55.0.69
PING 10.55.0.69 (10.55.0.69) 56(84) bytes of data.
64 bytes from 10.55.0.69: icmp_seq=1 ttl=60 time=13.6 ms
64 bytes from 10.55.0.69: icmp_seq=2 ttl=60 time=13.9 ms
64 bytes from 10.55.0.69: icmp_seq=3 ttl=60 time=13.9 ms
64 bytes from 10.55.0.69: icmp_seq=4 ttl=60 time=14.1 ms
^C
--- 10.55.0.69 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3002ms
rtt min/avg/max/mdev = 13.675/13.947/14.169/0.228 ms

在站点 8（最接近目标服务器）的橙色路由器上，我进行了此捕获（一切正常）：

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), capture size 262144 bytes
08:25:48.562284 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37119, seq 1, length 64
08:25:48.562389 IP 10.55.0.69 > 172.16.97.2: ICMP echo reply, id 37119, seq 1, length 64
08:25:49.563329 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37119, seq 2, length 64
08:25:49.563449 IP 10.55.0.69 > 172.16.97.2: ICMP echo reply, id 37119, seq 2, length 64
08:25:50.564448 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37119, seq 3, length 64
08:25:50.564593 IP 10.55.0.69 > 172.16.97.2: ICMP echo reply, id 37119, seq 3, length 64
08:25:51.565591 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37119, seq 4, length 64
08:25:51.565736 IP 10.55.0.69 > 172.16.97.2: ICMP echo reply, id 37119, seq 4, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel

现在，我正在发送相同的 ping，不分段指令和来自 172.16.97.2 服务器的 1300 字节

ping 10.55.0.69 -s 1300 -M do
PING 10.55.0.69 (10.55.0.69) 1300(1328) bytes of data.
From 10.255.0.73 icmp_seq=1 Frag needed and DF set (mtu = 894)
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
ping: local error: Message too long, mtu=894
^C
--- 10.55.0.69 ping statistics ---
8 packets transmitted, 0 received, +8 errors, 100% packet loss, time 7040ms

站点 7 上橙色路由器上的网络捕获

tcpdump -ni any host 172.16.97.2 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:32:27.586718 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37658, seq 1, length 1308
08:32:27.586751 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37658, seq 1, length 1308
08:32:27.598692 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 894), length 556
08:32:27.598705 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 894), length 556
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

站点 2 上橙色路由器上的网络捕获

tcpdump -ni any host 172.16.97.2 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:32:27.596503 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37658, seq 1, length 1308
08:32:27.596527 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 37658, seq 1, length 1308
08:32:27.596548 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 894), length 556
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

获取站点 7 上橙色路由器上的路由和 MTU

ip route get to 10.55.0.69
10.55.0.69 via 10.255.0.73 dev tunnel18 src 10.255.0.74 uid 0
cache expires 309sec mtu 1398

获取站点 2 上橙色路由器上的路由和 MTU

ip route get to 10.55.0.69
10.55.0.69 via 10.255.0.97 dev tunnel24 src 10.255.0.98 uid 0
cache expires 329sec mtu 894

让我们尝试将 ping 大小减小到 800 字节（因为缓存在上一个命令中过期 329sec mtu 894）

ping 10.55.0.69 -s 800 -M do
PING 10.55.0.69 (10.55.0.69) 800(828) bytes of data.
From 10.255.0.73 icmp_seq=1 Frag needed and DF set (mtu = 606)
ping: local error: Message too long, mtu=606
ping: local error: Message too long, mtu=606
ping: local error: Message too long, mtu=606
ping: local error: Message too long, mtu=606
^C
--- 10.55.0.69 ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4024ms

站点 7 上橙色路由器上的网络捕获

tcpdump -ni any host 172.16.97.2 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:39:42.735130 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 38450, seq 1, length 808
08:39:42.735143 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 38450, seq 1, length 808
08:39:42.746899 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 606), length 556
08:39:42.746911 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 606), length 556
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

ip route get to 10.55.0.69
10.55.0.69 via 10.255.0.73 dev tunnel18 src 10.255.0.74 uid 0
cache expires 582sec mtu 1398

站点 2 上橙色路由器上的网络捕获

tcpdump -ni any host 172.16.97.2 and icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
08:39:42.745058 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 38450, seq 1, length 808
08:39:42.745126 IP 172.16.97.2 > 10.55.0.69: ICMP echo request, id 38450, seq 1, length 808
08:39:42.745160 IP 10.255.0.73 > 172.16.97.2: ICMP 10.55.0.69 unreachable - need to frag (mtu 606), length 556
^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

ip route get to 10.55.0.69
10.55.0.69 via 10.255.0.97 dev tunnel24 src 10.255.0.98 uid 0
cache expires 582sec mtu 606

奇怪的是，MTU 从 894 减少到 606 字节

此外，在另一次测试中，我遇到了奇怪的事情（似乎 MTU 是动态更新的......）：

ping 10.55.0.69 -s 1010 -M do
PING 10.55.0.69 (10.55.0.69) 1010(1038) bytes of data.
1018 bytes from 10.55.0.69: icmp_seq=1 ttl=60 time=13.7 ms
1018 bytes from 10.55.0.69: icmp_seq=2 ttl=60 time=13.8 ms
1018 bytes from 10.55.0.69: icmp_seq=3 ttl=60 time=14.1 ms
1018 bytes from 10.55.0.69: icmp_seq=4 ttl=60 time=14.2 ms
1018 bytes from 10.55.0.69: icmp_seq=5 ttl=60 time=13.9 ms
From 10.255.0.73 icmp_seq=6 Frag needed and DF set (mtu = 966)
ping: local error: Message too long, mtu=966
ping: local error: Message too long, mtu=966
ping: local error: Message too long, mtu=966
ping: local error: Message too long, mtu=966
ping: local error: Message too long, mtu=966
^C
--- 10.55.0.69 ping statistics ---
11 packets transmitted, 5 received, +6 errors, 54% packet loss, time 10040ms
rtt min/avg/max/mdev = 13.759/14.000/14.297/0.226 ms

我的问题是：为什么 MTU 会这样动态减少？ 我知道 GRE 封装有开销，但这并不能解释为什么它减少了这么多我希望我提供足够的信息来解决我的问题:)

谢谢