对等连接重置-在k8s集群上使用驱动程序2.8.0和mongo 4.0.9

Question

We have been getting "Connection Reset by Peer" mongo errors in our setup. A description of the setup:

• mongo running as a replicaset in a k8s cluster on EKS
• clients (C#) running in the same k8s cluster on EKS
• mongo 4.0.9
• C# driver 2.8.0
• Connection pooling ON
• max idle time set to 10min (overrode default of 10s)
• max connection lifetime set to 10 min (overrode default of 10s)

We get these errors. We observed that if there is a series of calls, say 500 calls to do a key based select, there is no issue. Then we pause for 5 minutes, and repeat the test, the first time we get a "Connection Reset by Peer". Later, the test continues. This happens every time after pause.

This condition repeats with real users behavior, there may be spurts of activity and then a lull. As a consequence we keep getting "Connection reset by peer" at critical parts in the business workflow. On the client side, the solution is to perform defensive coding and repeat the call, but that's a change in many places.

Other combinations attempted:

• mongo 4.0.9
• C# driver 2.8.0
• Connection pooling ON
• max idle time 120min
• max connection lifetime 60min

However no change in the behavior.

It appears to us that while the TCP connection is closed on the server side, the client still thinks that it's a valid connection and attempts to use it, leading to this error.

Has anybody else faced such a situation? Any suggestions would be appreciated, happy to provide more information if needed.

Answer 1

I have a very similar issue with a cluster running on AKS. I managed to track this back to conntrack seeing (or thinking it is seeing) tcp retransmissions. Here is an example in which the client pod is 10.3.0.8 and the server pod 10.3.0.113, looking at the conntrack entries on the node running mongo:

conntrack -L|grep "10\\.3\\.0\\.113"|grep "10\\.3\\.0\\.88"

conntrack v1.4.3 (conntrack-tools): 1091 flow entries have been shown.
tcp      6 86398 ESTABLISHED src=10.3.0.88 dst=10.3.0.113 sport=34919 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=34919 [ASSURED] mark=0 use=1
tcp      6 86398 ESTABLISHED src=10.3.0.88 dst=10.3.0.113 sport=33389 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=33389 [ASSURED] mark=0 use=1
tcp      6 86390 ESTABLISHED src=10.3.0.88 dst=10.3.0.113 sport=39917 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=39917 [ASSURED] mark=0 use=1
tcp      6 51 TIME_WAIT src=10.3.0.88 dst=10.3.0.113 sport=36649 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=36649 [ASSURED] mark=0 use=1
tcp      6 298 ESTABLISHED src=10.3.0.88 dst=10.3.0.113 sport=35033 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=35033 [ASSURED] mark=0 use=1
tcp      6 299 ESTABLISHED src=10.3.0.88 dst=10.3.0.113 sport=44131 dport=27017 src=10.3.0.113 dst=10.3.0.88 sport=27017 dport=44131 [ASSURED] mark=0 use=1

You can see that there are some entries with very low timeouts (298/299 seconds) -- these started with 86400 seconds (/proc/sys/net/netfilter/nf_conntrack_tcp_timeout_established) but have been moved to 300 seconds (nf_conntrack_tcp_timeout_max_retrans). I am reasonably sure that this is the case because changing nf_conntrack_tcp_timeout_max_retrans changes the timeout value above.

I am at the present stage not sure why the retransmissions are occurring but it would be interesting to know if your problem is the same.

It can be worked around in my case by increasing nf_conntrack_tcp_timeout_max_retrans to > 10 minutes, or decreasing the mongo idle connection timeout to < 5 minutes.