Nodeport SFTP / SSH 使用 kube-proxy ipvs 时连接超时

Question

We are currently migrating from Docker Swarm to k8s (bare metal) and we cant reach the SFTP service in the pod. Service:

Name:                     mlflow-artifacts-store
Namespace:                mlflow
Labels:                   app.kubernetes.io/instance=mlflow-artifacts-store
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=mlflow-artifacts-store
                          helm.sh/chart=mlflow-artifacts-store-0.1.0
Annotations:              meta.helm.sh/release-name: mlflow-artifacts-store
                          meta.helm.sh/release-namespace: mlflow
Selector:                 app.kubernetes.io/instance=mlflow-artifacts-store,app.kubernetes.io/name=mlflow-artifacts-store
Type:                     NodePort
IP:                       10.233.24.136
Port:                     ssh  80/TCP
TargetPort:               22/TCP
NodePort:                 ssh  30001/TCP
Endpoints:                10.233.93.77:22
Session Affinity:         None
External Traffic Policy:  Cluster
Events:                   <none>

But I can't reach it even from the same server: (Timeout added for demonstration)

OpenSSH_7.4p1 Debian-10+deb9u7, OpenSSL 1.0.2u  20 Dec 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to 10.233.24.136 [10.233.24.136] port 30001.
debug1: fd 3 clearing O_NONBLOCK
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4p1 Debian-10+deb9u7
Connection timed out during banner exchange
Couldn't read packet: Connection reset by peer

The server itself is reachable from a different pod in the same namespace - therefore I guess it has probably something to do with the NodePort expose itself or the configuration.

Exposing the service with hostPort works, but I don't want to expose it like this. What do I miss?

Answer 1

So after discussing the whole thing on github the behaviour is intended. When using kube-proxy with ipvs the service is not reachable by the ipvs0 interface. https://github.com/kube.netes/kube.netes/issues/93674#issuecomment-669200021

ipvs is down by default and is just a dummy.network without traffic. In the documentation it's written like

The default for --nodeport-addresses is an empty list. This means that kube-proxy should consider all available.network interfaces for NodePort.

But ipvs0 is not reachable - this is intended. So the answer is: You should use a real address of the node.