"git clone" by ssh does sometimes not work on a virtual machine located at ESXi

Question

I am running a virtual machine (ubuntu 20.04) running on a ESXi 7.0 I want to clone a git repo via password less ssh access. Sometimes the git clone command just not respond, but sometimes everything runs smoothly for some hours. And right afterwards it's awkward again for hours.

I already figured out, I can reproduce the problem with the command ssh -T git@my-git-server.com I also tried to get some meaningful information by increasing the log level of ssh with: ssh -T -v -v -v git@my-git-server.com

the output is following (truncated):

...
debug1: Server accepts key: /home/gitlab-runner/.ssh/id_rsa RSA SHA256:klTmu54LL4yj61MM+isEbqfIhSq8wEJXZosw8eGDX9A
debug3: sign_and_send_pubkey: RSA SHA256:klTmu54LL4yj61MM+isEbqfIhSq8wEJXZosw8eGDX9A
debug3: sign_and_send_pubkey: signing using rsa-sha2-512 SHA256:klTmu54LL4yj61MM+isEbqfIhSq8wEJXZosw8eGDX9A
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to git-gccp.ds.mhie.com ([10.0.0.7]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Requesting no-more-sessions@openssh.com
debug3: send packet: type 80
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 80
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug3: receive packet: type 4
debug1: Remote: Forced command.
debug3: receive packet: type 4
debug1: Remote: Port forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: X11 forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: Agent forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: PTY allocation disabled.
debug3: receive packet: type 4
debug1: Remote: Forced command.
debug3: receive packet: type 4
debug1: Remote: Port forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: X11 forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: Agent forwarding disabled.
debug3: receive packet: type 4
debug1: Remote: PTY allocation disabled.
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env SUDO_GID
debug3: Ignored env SUDO_COMMAND
debug3: Ignored env SUDO_USER
debug3: Ignored env PWD
debug3: Ignored env LOGNAME
debug3: Ignored env HOME
debug1: Sending env LANG = C.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LS_COLORS
debug3: Ignored env TERM
debug3: Ignored env USER
debug3: Ignored env SHLVL
debug3: Ignored env PATH
debug3: Ignored env SUDO_UID
debug3: Ignored env MAIL
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768

But now I am stucked - No idea about the output. Whats going wrong here? I understand authentication worked, but what happened afterwards?

Would be cool to get some fresh thoughts and ideas. Did anyone ever faced similar problems?

I already tested this with dozens of different machines (Ubunut, RedHat, CentOS) on the same ESXi, all with same behavior. And from bare metal machines in the network we have rock solid connection to the git server. (which is located outside our network)

Answer 1

Just in case, double-check if firewall rules are not involved :

by default outgoing SSH / SCP from ESXi hosts is not allowed due to default firewall rules .
Before doing SSH/SCP from one ESXi host to another we need to allow SSH client in firewall rules of that ESXi host using:

• Login into vCenter and go to host from which SSH would be performed
• Go to Configure -> System -> Firewall
• Click " Edit " for incoming tab itself. Enable " SSH client "
• After this also if SSH is not working then do the same on destination host also.

Or, as in here :

esxcli network firewall ruleset set --enabled=true --ruleset-id=sshClient